gnutls_handshake() failed: Handshake failed in mTLS, works with SIMPLE

[apryiomka]

(NOTE: This is used to report product bugs:
To report a security vulnerability, please visit https://istio.io/about/security-vulnerabilities
To ask questions about how to use Istio, please visit https://discuss.istio.io)

Bug description
When switching to MUTUAL, secured gateway no longer works

curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:31047:10.0.3.159" "https://httpbin.example.com:31047/status/418" -k
* Added httpbin.example.com:31047:10.0.3.159 to DNS cache
* Hostname httpbin.example.com was found in DNS cache
*   Trying 10.0.3.159...
* Connected to httpbin.examplecom (10.0.3.159) port 31047 (#0)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* found 692 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* gnutls_handshake() failed: Handshake failed
* Closing connection 0
curl: (35) gnutls_handshake() failed: Handshake failed

** Expected Error ***
as per document

* TLSv1.3 (IN), TLS alert, unknown (628):
* OpenSSL SSL_read: error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required, errno 0

This works in SIMPLE mode

[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[x] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
[ ] Upgrade

Expected behavior

I have a secure gateway that looks like this:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: tls-default
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: MUTUAL
      credentialName: tls-certificate
    hosts:
    - '*'

and the secret:

apiVersion: v1
data:
  cacert: LS0tLS1CRUdJTiB***
  cert: LSBTHZDbTZaa***
  key: LS0tLS1***
kind: Secret
metadata:
  name: tls-certificate
  namespace: istio-system
type: Opaque

Steps to reproduce the bug

Version

istioctl version --remote
client version: 1.7.4
control plane version: 1.7.4
data plane version: 1.7.4 (9 proxies)

How was Istio installed?
IstioOperator
Environment where the bug was observed (cloud vendor, OS, etc)
AWS EKS

Edit:

The mTLS works when I make the request withing EKS cluster by providing the keys, but only 70% of the time. 2 out 3 requests would return the result, but 1 out of 3 would still fail with the same error:

curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:31047:10.0.3.159" "https://httpbin.example.com:31047/status/418" -k --cert client.com.cert.pem --key client.com.key.pem

But still the error without the key is not as per documentation.

Also it seems like it causes the aws-alb-controller ingress to return 502

[apryiomka]

I also tried to create a secret tls-certificate-cacert with cacert key, but that didn't make any difference

[howardjohn]

There are 100s of different curl TLS versions out there - our documentation cannot give the exact error message all of these may return, so I don't think we can expect that we can document every possible error message exactly as reported by all curl clients.

[istio-policy-bot]

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2021-03-19. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.