-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Slow TLS handshake when injecting the proxy with --tls=optional on an nginx-ingress pod #1880
Comments
Hi @jon-walton, I've tried to reproduce this using a self-signed certificate, and I haven't seen the delay (running Kubernetes in Docker for Mac Desktop). Creating the TLS secret:
Installing
Ingress configuration: apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: test-ingress
labels:
app.kubernetes.io/name: test-ingress
spec:
tls:
- hosts:
- example.com
secretName: example-tls
rules:
- host: example.com
http:
paths:
- path: /
backend:
serviceName: ingress-nginx-ingress-default-backend
servicePort: http Curl output (with timestamps, truncated):
Is there anything I've overlooked? I notice that your |
hi @hawkw Sorry for the slow reply... I'm also not able to reproduce this in docker for windows/mac. We're currently using kubernetes 1.12.1 deployed via acs-engine, so i'll see if I can track down if it's an azure/acs-engine thing or kubernetes version thing |
@jon-walton okay, thanks for the additional information. I'll look into reproducing the issue in Azure as well. |
Hmm, interesting --- attempting to reproduce this on Azure, I've noticed that, when injected with the proxy, the nginx ingress controller starts failing health checks and gets killed by Kubernetes. The new pod will also immediately fail health checks and get killed, so this continues endlessly. When the proxy isn't injected, the health checks seem to pass. I wonder if that's related? Update: When TLS isn't enabled on the injected proxy, nginx doesn't fail the health checks. This only happens with |
After reinjecting several times, my health check is no longer failing, and I can confirm that there's about a 3-second delay between the Client hello and the Server hello:
The linkerd proxy logs the following for that connection:
I wonder if the delay is due to the proxy having to parse the client hello up to SNI in order to determine whether or not it should terminate TLS for the connection? I don't think that should take three whole seconds, though... |
I've tested the same exact reproduction on GKE, and there doesn't seem to be any significant delay between the ClientHello and the ServerHello. So, this issue certainly appears to be Azure-specific. |
ok thanks. I'll see if I'm able to see a difference between using Azure CNI or Kubenet (probably late this week or next week) |
@jon-walton great, keep me posted! |
hi @hawkw Happy Christmas, here's a present for you 🤣 add and there's no longer a delay when injecting linkerd into the ingress controller pods with tls edit: it also fixed the weird behavior where the controller pods would fail healthchecks with
|
I spoke too soon on the healthchecks. The pods fell over when I put a few hundred rps though them. something for tomorrow i think 🤣 |
@hawkw So after further testing and having the ingress controllers running mostly ok for the past week, I believe the slowness is related to not setting let's put this issue on ice for now and I'll come back to it once #2012 is out of the way 👍 |
@jon-walton sounds good to me. I think #2012 will be resolved soon! |
i think this can be closed now. both |
@grampelberg do you think we should document that |
@hawkw it looks like there isn't a ton of support for |
Bug Report
What is the issue?
When using nginx-ingress and injecting the linkerd2 proxy with
--tls=optional
, any new TLS handshake originating externally from the cluster has a 2+ second delay between theClientHello
andServerHello
packets.How can it be reproduced?
--tls=optional
into the nginx ingress deployment# curl -v https://[ingress ip]
Test again without linkerd TLS on the nginx-ingress-controller:
There will not be a delay between
ClientHello
andServerHello
Logs, error output, etc
(If the output is long, please create a gist and
paste the link here.)
linkerd check
outputEnvironment
Possible solution
Additional context
The text was updated successfully, but these errors were encountered: