Istio egressgateway 16.1 TLS_error:_33554536:system_library:OPENSSL_internal:Connection_reset_by_peer

[psandhu79]

Bug Description

I followed this example
https://istio.io/latest/docs/tasks/traffic-management/egress/egress-gateway-tls-origination/

Everything works as expected however for one of the 3rd party api servers i connect to through the egress-gateway i get the following error

"upstream_transport_failure_reason":"TLS_error:_33554536:system_library:OPENSSL_internal:Connection_reset_by_peer",

almost the same as this issue #39077

the server in question i had previous issues with as well when using this config

previous egress gateway setup with ngnix sni proxy

I had the following options for ngnix sni proxy to make it work with the 3rd party api gateway

      proxy_ssl_name $host;
      proxy_http_version 1.1;
      proxy_ssl_server_name on;

However since switching to the preferred method it stopped working again

Version

client version: 1.9.1
control plane version: 1.16.1
data plane version: 1.16.1 (206 proxies)

Client Version: v1.24.0
Kustomize Version: v4.5.4
Server Version: v1.24.9-gke.2000

Additional Information

No response

[psandhu79]

Turned on debug but cannot see obvious error

jsonPayload: {
level: "debug"
msg: "[C82494][S12470871001166590339] upstream reset: reset reason: connection failure, transport failure reason: TLS error: 33554536:system library:OPENSSL_internal:Connection reset by peer"
scope: "envoy router"

If i exec into the egressgateway and curl the site curl -vvI https://dev.example.com

it gets all the TLS information ok

[liykfrank]

I'm experiencing the same issue.
It seems something wrong in TLS handshake. The server hello is missing. server side always return a Reset after client hello

[psandhu79]

looking in the trace logs the only thing i see wrong is
[C87372] ssl error occurred while read: WANT_READ

[GregHanson]

Was this working in a previous istio version or is 1.16.1 the first time you hit this problem? Also for TLS debugging you may need to turn on trace level logging in order to capture more of the error.

you may also want to check #42790 and #40680 to see if it's related

[psandhu79]

In the previous version we were using the ngnix sni proxy to foward on the https requests

I did turn on trace and posted the message above and other errors like
ssl read returns: -1", "scope":"envoy connection"}

the api gateway is using a wildcard cert from lets encrypt

If i run commands with the container of the egressgateway i can connect to the host on port 443

[psandhu79]

So I managed to fix it by looking this issue #32907

i had to add the sni at the end

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: echo-ext
spec:
  host: echo.dev.xxx
  trafficPolicy:
    portLevelSettings:
    - port:
        number: 80
      tls:
        mode: SIMPLE # initiates HTTPS when accessing echo.dev.xxx
        sni: echo.dev.xxx

Why some of the destination rules worked and this one didnt not sure.

[liykfrank]

In my case, it's caused by ECDSA certificates P-384 was not supported by Envoy by default.
So we have to enable P-384 with a EnvoyFilter
`
common_tls_context:

            tls_params:

              ecdh_curves: 

                - P-384`

[ldemailly]

@liykfrank thx, and you didn't hit the "only p256" exception that seems to be in the code (envoyproxy/envoy#10855 (comment)) ?