tls: first record does not look like a TLS handshake

[haarchri]

We are using the latest oauth-proxy in front of grafana ...

with the following configuration - the problem is we seeing every 30 seconds the following message in stdout/stderror:

2019/09/02 09:55:29 server.go:2753: http: TLS handshake error from 10.130.43.112:34584: tls: first record does not look like a TLS handshake
--
  | 2019/09/02 09:55:44 server.go:2753: http: TLS handshake error from 10.130.43.112:34842: tls: first record does not look like a TLS handshake
  | 2019/09/02 09:55:59 server.go:2753: http: TLS handshake error from 10.130.43.112:35122: tls: first record does not look like a TLS handshake
  | 2019/09/02 09:56:14 server.go:2753: http: TLS handshake error from 10.130.43.112:35390: tls: first record does not look like a TLS handshake
  | 2019/09/02 09:56:29 server.go:2753: http: TLS handshake error from 10.130.43.112:35664: tls: first record does not look like a TLS handshake
  | 2019/09/02 09:56:44 server.go:2753: http: TLS handshake error from 10.130.43.112:35926: tls: first record does not look like a TLS handshake
  | 2019/09/02 09:56:59 server.go:2753: http: TLS handshake error from 10.130.43.112:36214: tls: first record does not look like a TLS handshake
  | 2019/09/02 09:57:14 server.go:2753: http: TLS handshake error from 10.130.43.112:36484: tls: first record does not look like a TLS handshake
  | 2019/09/02 09:57:29 server.go:2753: http: TLS handshake error from 10.130.43.112:36764: tls: first record does not look like a TLS handshake

        - args:
            - '-https-address=:8443'
            - '-http-address='
            - '-email-domain=*'
            - '-client-id=system:serviceaccount:test:grafana'
            - '-upstream=http://localhost:3000'
            - '-provider=openshift'
            - >-
              -openshift-sar={"namespace": "test", "verb": "list", "resource":
              "services"}
            - '-tls-cert=/etc/tls/private/tls.crt'
            - '-tls-key=/etc/tls/private/tls.key'
            - >-
              -client-secret-file=/var/run/secrets/kubernetes.io/serviceaccount/token
            - '-cookie-secret-file=/etc/proxy/secrets/session_secret'
            - '-skip-auth-regex=^/metrics,/api/datasources,/api/dashboards'
          image: 'dockerhub.io/openshift/oauth-proxy:latest'

[mriedmann]

Just a hint:
Check if your Routes are set to "Re-encrypt" instead of "Edge".

Was my problem seconds ago.

[gfvirga]

We are having the same problem on openshift4.2
I can't seem to change from passthrough to reencrypt without changing the operator to unmanaged. Shouldn't be the right approach.

[stlaz]

@gfvirga Which operator would that be?

This message appears when you're trying to reach an HTTPS-only endpoint by a plain HTTP request, so it's probably not a problem in the proxy, but whatever that's trying to reach it HTTP-first.

[stlaz]

Note that the above also explains why setting the route from Edge (from docs - Because TLS is terminated at the router, connections from the router to the endpoints over the internal network are not encrypted.) to Re-encrypt would fix the issue as you're effectively switching from HTTP to HTTPS requests being forwarded to the proxy.

[gfvirga]

We are getting ":500 Internal Error" when we try to login to the routes: alertmanager-main, grafana and prometheus-k8s.
I noticed the TLS handshake error in the oauth-openshift pod, but I am not sure how to troubleshoot this one. First I thought it was proxy and then I thought it was the ingress SSL cert, but neither are a problem from the oauth-openshift terminal. There were no errors in the alermanager, grafana and prometheus pods.

This log is on oauth-openshift pod:

I1017 00:11:10.507334 1 log.go:172] http: TLS handshake error from 10.130.0.1:47784: EOF
I1017 00:18:13.698951 1 log.go:172] http: TLS handshake error from 10.128.0.1:35522: remote error: tls: bad certificate

This curl showing that it is not going through the proxy.

sh-4.2# curl -I 10.130.0.1
HTTP/1.0 503 Service Unavailable
Pragma: no-cache
Cache-Control: private, max-age=0, no-cache, no-store
Connection: close
Content-Type: text/html

sh-4.2# curl -I https://alertmanager-main-openshift-monitoring.apps.ose.com
HTTP/1.1 403 Forbidden
Set-Cookie: _oauth_proxy=; Path=/; Domain=alertmanager-main-openshift-monitoring.apps.ose.com; Expires=Thu, 17 Oct 2019 12:33:37 GMT; HttpOnly; Secure
Date: Thu, 17 Oct 2019 13:33:37 GMT
Content-Type: text/html; charset=utf-8
Set-Cookie: 4e2f052a85d89824d1855ef94e095b2b=a770a836fef9acd6788dafd5437d79b5; path=/; HttpOnly; Secure

[stlaz]

@gfvirga Please open a Bug for the Auth component here: https://bugzilla.redhat.com/enter_bug.cgi?product=OpenShift%20Container%20Platform.

Include logs from the oauth-proxy, oauth-server and, your IdP config (from OAuth object) and at least the output of oc get clusteroperator -o json. Either me or my colleagues will have a look.

I'm going to close this issue since the behavior is expected when only -https-address=:8443 is set but the proxy is getting pure HTTP requests.

[gfvirga]

@stlaz, the error I was getting is related to the CAbundle not being added to the sidecar proxy pods.
openshift/cluster-monitoring-operator#526
openshift/cluster-logging-operator#261