Not able to access any HTTPS external services using Istio

[anoopl]

We have Istio cluster running on 1.1.6 and 1.1.4
We have similar issue on both the clusters that not able to access any external HTTPS url like Google. We do not have any Service Entry and mTLS is disabled cluster wise. There is no logs on the istio-proxy side car.

 curl https://www.google.com --verbose
* Rebuilt URL to: https://www.google.com/
*   Trying 172.217.6.68...
* TCP_NODELAY set
* Connected to www.google.com (172.217.6.68) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0
curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

Where should we look for some debug information. Any help on this would be great

We run Kuberenetes v1.13.2 with Weavenet CNI

We have this issue only with pods that have istio sidecar

[GregHanson]

@anoopl can you try deploying the sample sleep service and verify the behavior is the same from there?

kubectl apply -f samples/sleep/sleep.yaml

Wait for the sleep pod to finish creating, then run the following:

$ kubectl exec -it $(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name}) -c sleep -- curl -isv https://www.google.com
* Rebuilt URL to: https://www.google.com/
*   Trying 216.58.194.68...
* TCP_NODELAY set
* Connected to www.google.com (216.58.194.68) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=www.google.com
*  start date: Apr 30 09:21:28 2019 GMT
*  expire date: Jul 23 09:03:00 2019 GMT
*  subjectAltName: host "www.google.com" matched cert's "www.google.com"
*  issuer: C=US; O=Google Trust Services; CN=Google Internet Authority G3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55c2ba9bd5e0)
> GET / HTTP/2
> Host: www.google.com
> User-Agent: curl/7.60.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200 
HTTP/2 200 
< date: Tue, 21 May 2019 16:22:28 GMT
date: Tue, 21 May 2019 16:22:28 GMT
< expires: -1
expires: -1
< cache-control: private, max-age=0
cache-control: private, max-age=0
< content-type: text/html; charset=ISO-8859-1
content-type: text/html; charset=ISO-8859-1
< p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
< server: gws
server: gws
< x-xss-protection: 0
x-xss-protection: 0
< x-frame-options: SAMEORIGIN
x-frame-options: SAMEORIGIN
< set-cookie: 1P_JAR=2019-05-21-16; expires=Thu, 20-Jun-2019 16:22:28 GMT; path=/; domain=.google.com
set-cookie: 1P_JAR=2019-05-21-16; expires=Thu, 20-Jun-2019 16:22:28 GMT; path=/; domain=.google.com
< set-cookie: NID=183=1ORfSv-s7BzXI_RRKCWuLT-AS9PIdAQdXxZfLogcJslvCjt4iAyseMiY7iI3ZrDBWijVHu7ApTKMPNrsQotfu3Boi43g5FEZOTyH4B_CLSqMO1JYiZ3Ej2wWlHCcRtyy-NChJOPRnVrEbT5qc_rg_dwsAb94YNXAjIiZz__wKqU; expires=Wed, 20-Nov-2019 16:22:28 GMT; path=/; domain=.google.com; HttpOnly
set-cookie: NID=183=1ORfSv-s7BzXI_RRKCWuLT-AS9PIdAQdXxZfLogcJslvCjt4iAyseMiY7iI3ZrDBWijVHu7ApTKMPNrsQotfu3Boi43g5FEZOTyH4B_CLSqMO1JYiZ3Ej2wWlHCcRtyy-NChJOPRnVrEbT5qc_rg_dwsAb94YNXAjIiZz__wKqU; expires=Wed, 20-Nov-2019 16:22:28 GMT; path=/; domain=.google.com; HttpOnly
< alt-svc: quic=":443"; ma=2592000; v="46,44,43,39"
alt-svc: quic=":443"; ma=2592000; v="46,44,43,39"
< accept-ranges: none
accept-ranges: none
< vary: Accept-Encoding
vary: Accept-Encoding

 . . . .<HTML> . . .  .

I am using Istio 1.1.6 without any ServiceEntries. This check will help decide if it is a problem with config (i.e. VirtualService rules or istio deployment configuration) or a problem with your app image and the packages currently installed

[anoopl]

@GregHanson I tried that now, got the same error

  kubectl exec -it $(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name}) -c sleep -- curl -isv https://google.com
* Rebuilt URL to: https://google.com/
*   Trying 172.217.5.110...
* TCP_NODELAY set
* Connected to google.com (172.217.5.110) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number
* stopped the pause stream!
* Closing connection 0
command terminated with exit code 35

[GregHanson]

based on 2607:f8b0:4005:808::200e, your sleep service appears to be using IPv6
@sbezverk I know you have been working with IPv6, any thoughts?

Also, do you get any different behavior with a ServiceEntry defined?
https://github.com/istio/istio/blob/master/tests/e2e/tests/pilot/testdata/networking/v1alpha3/service-entry-google.yaml

[anoopl]

@GregHanson I tried the ServiceEntry already and tried it again now. It's the same error with ServiceEntry as well.

[anoopl]

To Post a most recent output. It's IPV4:

 kubectl exec -it $(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name}) -c sleep -- curl -isv https://google.com
* Rebuilt URL to: https://google.com/
*   Trying 172.217.5.110...
* TCP_NODELAY set
* Connected to google.com (172.217.5.110) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number
* stopped the pause stream!
* Closing connection 0
command terminated with exit code 35

I think only at that point it tried IPV6 when IPV4 failed

[sbezverk]

Final ipv6 bits recently went into master, I am not sure if they were cherrypicked into 1.1.4 or .6
Cluster will listen on ipv6 only if hostname command in container return ipv6 as a primary IP address.

I would start checking sleep why it listens on ipv6 instead of ipv4.

I cannot repro this issue since I am travelling this week.

Please send steps to repro, I will do it up on my return.

[anoopl]

@sbezverk @GregHanson
To Post a most recent output. It's IPV4:

 kubectl exec -it $(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name}) -c sleep -- curl -isv https://google.com
* Rebuilt URL to: https://google.com/
*   Trying 172.217.5.110...
* TCP_NODELAY set
* Connected to google.com (172.217.5.110) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number
* stopped the pause stream!
* Closing connection 0
command terminated with exit code 35

I think only at that point it tried IPV6 when IPV4 failed

 # hostname
sleep-754684654f-bs252
/ # cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.39.208.1     sleep-754684654f-bs252

[GregHanson]

Double-check the output from these commands:

$ kubectl get Meshpolicy default -o yaml
apiVersion: authentication.istio.io/v1alpha1
...
...
      mode: PERMISSIVE

and

$ kubectl get configmap istio -n istio-system -o yaml | grep -o "mode: ALLOW_ANY"
mode: ALLOW_ANY

Make sure you are running with the expected configurations and did not have anything lying around from a previous deployment

Also, enable access logs if they are not already enabled (see here). Perform the curls from both apps again and check the corresponding sidecar access logs for an entry

Additionally, check what rules you currently have created:

istioctl get all --all-namespaces

^^make sure the dates are not older than your current deployment. Can also append -o yaml to the istioctl command to get the rules in yaml format to check for any rules that might be creating conflicts on port 443 or for hostname google

[anoopl]

@GregHanson
mTLS Mesh policy

spec:
  peers:
  - mtls:
      mode: PERMISSIVE

 kubectl get configmap istio -n istio-system -o yaml | grep -o "mode: ALLOW_ANY"
mode: ALLOW_ANY

istioctl get all --all-namespaces
Command "get" is deprecated, Use `kubectl get` instead (see https://kubernetes.io/docs/tasks/tools/install-kubectl)
NAME      KIND                                          NAMESPACE   AGE
default   MeshPolicy.authentication.istio.io.v1alpha1               4d

DESTINATION-RULE NAME   HOST                                             SUBSETS   NAMESPACE      AGE
istio-policy            istio-policy.istio-system.svc.cluster.local                istio-system   4d
istio-telemetry         istio-telemetry.istio-system.svc.cluster.local             istio-system   4d

So everything looks good. I have debug mode enabled for the proxies, I do not see any logs on istio-proxy logs related to this

[anoopl]

It's not just google.com. I have same error with any HTTPS url

 kubectl exec -it $(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name}) -c sleep -- curl -isv https://www.bbc.com
* Rebuilt URL to: https://www.bbc.com/
*   Trying 199.232.8.81...
* TCP_NODELAY set
* Connected to www.bbc.com (199.232.8.81) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number
* stopped the pause stream!
* Closing connection 0
command terminated with exit code 35

[mabushey]

If you install Istio with the amazing Bonzai operator, not only will you have a cleaner/better install of Istio, but outgoing https requests from INSIDE the cluster will work.

✗ kubectl apply -f sleep.yaml                                                                                                                                                                                                                                   
...
 ✗ kubectl exec -it $(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name}) -c sleep -- curl -isv https://www.google.com                                                                                                                             
* Rebuilt URL to: https://www.google.com/                                                                                                                                                                                                                                                 
*   Trying 216.58.193.68...                                                                                                                                                                                                                                                               
* TCP_NODELAY set                                                                                                                                                                                                                                                                         
* Connected to www.google.com (216.58.193.68) port 443 (#0) 
* Connected to www.google.com (216.58.193.68) port 443 (#0)                                                                                                                                                                                                                               
* ALPN, offering h2                                                                                                                                                                                                                                                                       
* ALPN, offering http/1.1                                                                                                                                                                                                                                                                 
* successfully set certificate verify locations:                                                                                                                                                                                                                                          
*   CAfile: /etc/ssl/certs/ca-certificates.crt                                                                                                                                                                                                                                            
  CApath: none                                                                                                                                                                                                                                                                            
* TLSv1.2 (OUT), TLS handshake, Client hello (1):                                                                                                                                                                                                                                         
* TLSv1.2 (IN), TLS handshake, Server hello (2):                                                                                                                                                                                                                                          
* TLSv1.2 (IN), TLS handshake, Certificate (11):                                                                                                                                                                                                                                          
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):                                                                                                                                                                                                                                  
* TLSv1.2 (IN), TLS handshake, Server finished (14):                                                                                                                                                                                                                                      
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):                                                                                                                                                                                                                                 
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):                                                                                                                                                                                                                                     
* TLSv1.2 (OUT), TLS handshake, Finished (20):                                                                                                                                                                                                                                            
* TLSv1.2 (IN), TLS change cipher, Client hello (1):                                                                                                                                                                                                                                      
* TLSv1.2 (IN), TLS handshake, Finished (20):                                                                                                                                                                                                                                             
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305                                                                                                                                                                                                                            
* ALPN, server accepted to use h2                                                                                                                                                                                                                                                         
* Server certificate:                                                                                                                                                                                                                                                                     
*  subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=www.google.com                                                                                                                                                                                                         
*  start date: Apr 30 09:21:28 2019 GMT                                                                                                                                                                                                                                                   
*  expire date: Jul 23 09:03:00 2019 GMT 
*  subjectAltName: host "www.google.com" matched cert's "www.google.com"
*  issuer: C=US; O=Google Trust Services; CN=Google Internet Authority G3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x559125d945e0)
> GET / HTTP/2
> Host: www.google.com
> User-Agent: curl/7.60.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200
HTTP/2 200
< date: Thu, 23 May 2019 00:21:49 GMT
date: Thu, 23 May 2019 00:21:49 GMT
< expires: -1
expires: -1
< cache-control: private, max-age=0
cache-control: private, max-age=0
< content-type: text/html; charset=ISO-8859-1
content-type: text/html; charset=ISO-8859-1
< p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
< server: gws
server: gws
< x-xss-protection: 0
x-xss-protection: 0
< x-frame-options: SAMEORIGIN
x-frame-options: SAMEORIGIN
< set-cookie: 1P_JAR=2019-05-23-00; expires=Sat, 22-Jun-2019 00:21:49 GMT; path=/; domain=.google.com
set-cookie: 1P_JAR=2019-05-23-00; expires=Sat, 22-Jun-2019 00:21:49 GMT; path=/; domain=.google.com
< set-cookie: NID=184=DIhLgCnPg0cskBTw4iNsMzoAzW7LjbM_HFafZlxo5Z3zQfhlzMOiz0JJTbf80FdBNwfxyG0TrfucUZM09ZRSxePYPsUr3Hs9WPjltt4yAtHXVYH4IsxuBzvU3eUHSN1JpK3dRHv5gUQ7gC-AffV2AjDWO2mpn6GkHcVdqkYviVE; expires=Fri, 22-Nov-2019 00:21:49 GMT; path=/; domain=.google.com; HttpOnly
set-cookie: NID=184=DIhLgCnPg0cskBTw4iNsMzoAzW7LjbM_HFafZlxo5Z3zQfhlzMOiz0JJTbf80FdBNwfxyG0TrfucUZM09ZRSxePYPsUr3Hs9WPjltt4yAtHXVYH4IsxuBzvU3eUHSN1JpK3dRHv5gUQ7gC-AffV2AjDWO2mpn6GkHcVdqkYviVE; expires=Fri, 22-Nov-2019 00:21:49 GMT; path=/; domain=.google.com; HttpOnly
< alt-svc: quic=":443"; ma=2592000; v="46,44,43,39"
alt-svc: quic=":443"; ma=2592000; v="46,44,43,39"
< accept-ranges: none
accept-ranges: none
< vary: Accept-Encoding
vary: Accept-Encoding

<
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><head><meta content="Search the world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for."
...

I'm running Istio 1.1.2.

[chr15murray]

I've been having the same issue and finally resolved it today with help from this link..
https://discuss.istio.io/t/serviceentry-for-https-on-httpbin-org-resulting-in-connect-cr-srvr-hello-using-curl/2044/3

For me, I was deploying a dedicated ingress gateway for my application and as part of that I added a port called 'http-port443'.
This in tern was creating a service (as part of the ingress deployment) and I then hit the same problem described in the above link.
My fix was to rename the port to 'https-port443' and now egress is working correctly.

Please also see here for an alternative workaround: #13914 (comment)

This leads me to think that the problem is with how the sidecar forwards https traffic to the egress but I don't know enough about this to dig deeper and confirm if this is a bug or not.

Hope this helps

[anoopl]

@chr15murray Thanks for the suggestion.
I was trying to debug the same way:

istioctl  proxy-status
NAME                                                   CDS        LDS        EDS               RDS          PILOT                            VERSION
istio-egressgateway-7d66d8fb56-l2pp5.istio-system      SYNCED     SYNCED     SYNCED (100%)     NOT SENT     istio-pilot-8565c8cbfb-tfg24     1.1.3
istio-ingressgateway-85c5f6f5df-vg4j7.istio-system     SYNCED     SYNCED     SYNCED (100%)     NOT SENT     istio-pilot-8565c8cbfb-tfg24     1.1.3
sleep-754684654f-n74s8.test-ns                         SYNCED     SYNCED     SYNCED (50%)      SYNCED       istio-pilot-8565c8cbfb-tfg24     1.1.3

But I am not able to get the proxy-config cluster/route:

istioctl -n test-ns proxy-config cluster sleep-754684654f-n74s8.test-ns
Error: failed to execute command on envoy: error execing into sleep-754684654f-n74s8/test-ns istio-proxy container: command terminated with exit code 255

istioctl -n test-ns proxy-config routes sleep-754684654f-n74s8.test-ns --name 443  -o json
Error: failed to execute command on envoy: error execing into sleep-754684654f-n74s8/test-ns istio-proxy container: command terminated with exit code 255

I am using the sleep.yaml from istio samples,
Do I have to run some other image for this to work? Also i see SYNCED (50%) on proxy-status. Is that fine?

[anoopl]

I was able to get the proxy-config route working on another cluster.
I found the issue as same as discussed here. https://discuss.istio.io/t/serviceentry-for-https-on-httpbin-org-resulting-in-connect-cr-srvr-hello-using-curl/2044/3

When i do : istioctl -n ist-stg-duck-stg proxy-config routes sleep-754684654f-bnffk.<namespace> --name 443 -o json
It showed VirtualHosts to other 3 services like:

    {
        "name": "443",
        "virtualHosts": [
            {
                "name": "svc1.svc.cluster.local:443",

On checking this services we found that this services using name HTTP for 443 port like:

  clusterIP: xx.xx.xx.xx
  ports:
  - name: http
    port: 443
    protocol: TCP
    targetPort: 443

I renamed it to HTTPS and checked the route which showed:

    {
        "name": "443",
        "virtualHosts": [
            {
                "name": "allow_any",
                "domains": [
                    "*"
                ],
                "routes": [
                    {
                        "match": {
                            "prefix": "/"
                        },
                        "route": {
                            "cluster": "PassthroughCluster"
                        },
                        "perFilterConfig": {
                            "mixer": {
                                "disable_check_calls": true,
                                "forward_attributes": {
                                    "attributes": {}
                                },
                                "mixer_attributes": {
                                    "attributes": {}
                                }
                            }
                        }
                    }
                ]
            }
        ],
        "validateClusters": false
    }
]

After this curl https://www.google.com worked all like expected.

We have not enabled istio on this namespace. We do not know the reason for this bug.
We are glad that at least we have a work around. However this can happen again if some of our client deploys a service with name HTTP for 443.

Thanks @chr15murray helping me on debugging this. Thanks to everyone.