SSL setup fails with: CONNECT_CR_SRVR_HELLO:wrong version number

[cjohansen]

NGINX Ingress controller version: 0.20.0

Kubernetes version (use kubectl version): 1.10.3

Environment: AWS

• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): Debian Jessie
• Kernel (e.g. uname -a): Linux ip-x-y-z-a 4.4.121-k8s Basic structure  #1 SMP Sun Mar 11 19:39:47 UTC 2018 x86_64 GNU/Linux
• Install tools: Kops

What happened:

Followed the ingress TLS instructions: put my wildcard certificate key and crt files in a secret, and annotated my ingress thusly:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-web-service
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.org/websocket-services: "my-service"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
  tls:
    - hosts:
        - myservice.mydomain.tld
      secretName: mydomain-tld-cert
  rules:
    - host: myservice.mydomain.tld
      http:
        paths:
          - path: /
            backend:
              serviceName: my-service
              servicePort: my-service-api-port

After applying this configuration, my service fails:

curl -vvv https://myservice.mydomain.tld
* Rebuilt URL to: https://myservice.mydomain.tld/
*   Trying 66.66.66.66...
* TCP_NODELAY set
* Connected to myservice.mydomain.tld (66.66.66.66) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number
* stopped the pause stream!
* Closing connection 0
curl: (35) error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number

Additionally, I see encrypted garbage in the logs:

100.106.0.0 - [100.106.0.0] - - [12/Dec/2018:13:01:39 +0000] "\x16\x03\x01\x00\xDE\x01\x00\x00\xDA\x03\x03\xAB\x9E\x19\xC2\xE3\xE3\xC6bJy\x80\xCA\x0F^\x7F<\xF5\x0B/t\x1CG\x9Fh}\xDD\xB6aJ\x98\xAE\xBC\x00\x00T\xCC\xA9\xCC\xA8\xCC\xAA\xC00\xC0,\xC0(\xC0$\xC0\x14\xC0" 400 157 "-" "-" 0 0.045 [] - - - - 9420db1697d32fc5704f17cd58b1d66b

What you expected to happen:

A valid HTTP response to come back.

How to reproduce it (as minimally and precisely as possible):

I can't really give you my certificate key, so I'm hoping this will help without explicit reproduction instructions.

Anything else we need to know:

I peeked at the generated Nginx config, and it looks right. I pulled down the pem-files generated by the ingress controller and copied the SSL part of the Nginx config, and ran it all in a local Nginx, and it worked as expected, so the key and certificate is alright.

[aledbf]

100.106.0.0 - [100.106.0.0] - - [12/Dec/2018:13:01:39 +0000] "\x16\x03\x01\x00\xDE\x01\x00\x00\xDA\x03\x03\xAB\x9E\x19\xC2\xE3\xE3\xC6bJy\x80\xCA\x0F^\x7F<\xF5\x0B/t\x1CG\x9Fh}\xDD\xB6aJ\x98\xAE\xBC\x00\x00T\xCC\xA9\xCC\xA8\xCC\xAA\xC00\xC0,\xC0(\xC0$\xC0\x14\xC0" 400 157 "-" "-" 0 0.045 [] - - - - 9420db1697d32fc5704f17cd58b1d66b

That means you are sending TCP content (HTTPS) to an HTTP port. Are you using an ELB? Please check the port mappings

[cjohansen]

Holy moly, I'm so sorry for this. And thanks a lot for the pointer! Turns out the container configuration for the ingress controller had two ports: socket-http and socket-https, and I'd misspelled one in the service config, effectively routing all traffic to the http port. FACEPALM. Thanks again!

[aledbf]

curl -i -H 'Host: myservice.mydomain.tld' https://xyz.elb.eu-west-1.amazonaws.com

That is not valid for https request (actually a request with SNI). You need to run something like:

curl -kvL --resolve myservice.mydomain.tld:443:x.x.x.x https://xyz.elb.eu-west-1.amazonaws.com

Using the host header in TLS connections is not going to fulfill that requirement
https://curl.haxx.se/mail/archive-2015-01/0042.html

[cjohansen]

Hah, you were too quick for me. I realised my mistake in writing that second comment, and deleted it. Anyway, thanks for the heads up!

[anthonybudd]

That means you are sending TCP content (HTTPS) to an HTTP port. Are you using an ELB? Please check the port mappings

Could you elaborate a bit more? What exactly do you mean and what would you suggest to solve this issue? Is the problem in the ingress, load balancer or service?

[ismailyenigul]

Hello,

I have a similar issue but this time the traffic initiated from a pod in the same k8s cluster with ingress-nginx

I am running ingress-nginx on digital ocean k8s managed cluster.
Configured ingres-nginx for a service
here is the service defination.

apiVersion: v1
kind: Service
metadata:
  annotations:
   
    service.beta.kubernetes.io/do-loadbalancer-algorithm: round_robin
    service.beta.kubernetes.io/do-loadbalancer-certificate-id: 4xxy134651-c97c-41a2-93bd-4baef1a63ff4
    service.beta.kubernetes.io/do-loadbalancer-healthcheck-path: /healthz
    service.beta.kubernetes.io/do-loadbalancer-healthcheck-protocol: http
    service.beta.kubernetes.io/do-loadbalancer-protocol: http
    service.beta.kubernetes.io/do-loadbalancer-redirect-http-to-https: "true"
    service.beta.kubernetes.io/do-loadbalancer-tls-ports: "443"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  name: ingress-nginx
  namespace: ingress-nginx
spec:
  clusterIP: 10.245.23.188
  externalTrafficPolicy: Cluster
  ports:
  - name: http
    nodePort: 31645
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    nodePort: 30696
    port: 443
    protocol: TCP
    targetPort: 80
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  sessionAffinity: None
  type: LoadBalancer

If I access this URL from out of k8s cluster it works fine.

But I access from a pod in the eks cluster, It fails
It seems that Load balancer HTTPS termination does not work for the sources inside the k8s.
and I hit the following error. wrong version number error happens while requesting data from HTTP service with HTTPS request.

$ curl  https://dev.mydomain.com  -v
*   Trying 138.68.127.x:443...
* Connected to dev-auth.appcircle.io (138.68.127.x) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

Is there any config/parameter to allow this HTTPS termination inside k8s?

well, you may ask why I don't connect to service over port 80!
The problem is here I generate a token and app runs behind on this service sign the token basd on called URL for that reason I have to access over HTTPS url.

[ismailyenigul]

I see that this problem is related to DigitalOcean kubernetes
the fix is setting hostname for the ingress service.

The details are at digitalocean/DOKS#8

https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/master/docs/controllers/services/annotations.md#servicebetakubernetesiodo-loadbalancer-hostname

[rdp]

For followers I once got this message downloading a file from one computer but not another, on the same network. Hunch/suspicion: anti virus or somehow corporate something or other got in the way...

[chance2021]

100.106.0.0 - [100.106.0.0] - - [12/Dec/2018:13:01:39 +0000] "\x16\x03\x01\x00\xDE\x01\x00\x00\xDA\x03\x03\xAB\x9E\x19\xC2\xE3\xE3\xC6bJy\x80\xCA\x0F^\x7F<\xF5\x0B/t\x1CG\x9Fh}\xDD\xB6aJ\x98\xAE\xBC\x00\x00T\xCC\xA9\xCC\xA8\xCC\xAA\xC00\xC0,\xC0(\xC0$\xC0\x14\xC0" 400 157 "-" "-" 0 0.045 [] - - - - 9420db1697d32fc5704f17cd58b1d66b

That means you are sending TCP content (HTTPS) to an HTTP port. Are you using an ELB? Please check the port mappings

100.106.0.0 - [100.106.0.0] - - [12/Dec/2018:13:01:39 +0000] "\x16\x03\x01\x00\xDE\x01\x00\x00\xDA\x03\x03\xAB\x9E\x19\xC2\xE3\xE3\xC6bJy\x80\xCA\x0F^\x7F<\xF5\x0B/t\x1CG\x9Fh}\xDD\xB6aJ\x98\xAE\xBC\x00\x00T\xCC\xA9\xCC\xA8\xCC\xAA\xC00\xC0,\xC0(\xC0$\xC0\x14\xC0" 400 157 "-" "-" 0 0.045 [] - - - - 9420db1697d32fc5704f17cd58b1d66b

That means you are sending TCP content (HTTPS) to an HTTP port. Are you using an ELB? Please check the port mappings

Thanks @aledbf ! My issue happened when I used curl to communicate with vault. It turns out that my vault was disabled tls. I just redeployed the vault with tls enabled and it works perfectly now! Thanks