TLS not working #4227
Comments
nmengin
added
area/provider/k8s/ingress
priority/P1
|
I read |
|
@athoune yes. Are you asking a question? If you read the whole post, the secret exists -- proof at the bottom of the post. It's just Traefik doesn't see it or can't find it. |
|
@aug70 Does the certificate actually work? or is it just the error message you are encountering? There is a known issue: (#3667) That references that error message. If you are finding that the certificate is not being loaded at all by traefik, I would check the RBAC that your traefik pod serviceAccount runs with. It is possible that traefik may not have the correct permissions to access those secrets, even if the serviceAccount you use to run |
|
@dtomcej The certificate is legit and I use it in many other places but it's a wild card certificate 🤔
Works just fine without https |
|
@aug70 Can you try enabling the debug logs? I want to see the events provided by k8s. Also of note: Is there a particular reason you are patching the secret type? I don't think it should matter, but can you try without patching the type, and leaving it Thanks! |
time="2018-12-08T05:56:53Z" level=info msg="Using TOML configuration file /config/traefik.toml"
time="2018-12-08T05:56:53Z" level=info msg="No tls.defaultCertificate given for https: using the first item in tls.certificates as a fallback."
time="2018-12-08T05:56:53Z" level=info msg="Traefik version v1.7.5 built on 2018-12-03_11:01:00AM"
time="2018-12-08T05:56:53Z" level=debug msg="Global configuration loaded {\"LifeCycle\":{\"RequestAcceptGraceTimeout\":0,\"GraceTimeOut\":10000000000},\"GraceTimeOut\":0,\"Debug\":false,\"CheckNewVersion\":true,\"SendAnonymousUsage\":false,\"AccessLogsFile\":\"\",\"AccessLog\":null,\"TraefikLogsFile\":\"\",\"TraefikLog\":null,\"Tracing\":null,\"LogLevel\":\"DEBUG\",\"EntryPoints\":{\"http\":{\"Address\":\":80\",\"TLS\":null,\"Redirect\":{\"entryPoint\":\"https\"},\"Auth\":null,\"WhitelistSourceRange\":null,\"WhiteList\":null,\"Compress\":false,\"ProxyProtocol\":null,\"ForwardedHeaders\":{\"Insecure\":true,\"TrustedIPs\":null}},\"https\":{\"Address\":\":443\",\"TLS\":{\"MinVersion\":\"\",\"CipherSuites\":null,\"Certificates\":[{\"CertFile\":\"/ssl/tls.crt\",\"KeyFile\":\"/ssl/tls.key\"}],\"ClientCAFiles\":null,\"ClientCA\":{\"Files\":null,\"Optional\":false},\"DefaultCertificate\":{\"CertFile\":\"/ssl/tls.crt\",\"KeyFile\":\"/ssl/tls.key\"},\"SniStrict\":false},\"Redirect\":null,\"Auth\":null,\"WhitelistSourceRange\":null,\"WhiteList\":null,\"Compress\":false,\"ProxyProtocol\":null,\"ForwardedHeaders\":{\"Insecure\":true,\"TrustedIPs\":null}}},\"Cluster\":null,\"Constraints\":[],\"ACME\":null,\"DefaultEntryPoints\":[\"http\",\"https\"],\"ProvidersThrottleDuration\":2000000000,\"MaxIdleConnsPerHost\":200,\"IdleTimeout\":0,\"InsecureSkipVerify\":false,\"RootCAs\":null,\"Retry\":null,\"HealthCheck\":{\"Interval\":30000000000},\"RespondingTimeouts\":null,\"ForwardingTimeouts\":null,\"AllowMinWeightZero\":false,\"KeepTrailingSlash\":false,\"Web\":null,\"Docker\":null,\"File\":null,\"Marathon\":null,\"Consul\":null,\"ConsulCatalog\":null,\"Etcd\":null,\"Zookeeper\":null,\"Boltdb\":null,\"Kubernetes\":{\"Watch\":true,\"Filename\":\"\",\"Constraints\":[],\"Trace\":false,\"TemplateVersion\":0,\"DebugLogGeneratedTemplate\":false,\"Endpoint\":\"\",\"Token\":\"\",\"CertAuthFilePath\":\"\",\"DisablePassHostHeaders\":false,\"EnablePassTLSCert\":false,\"Namespaces\":null,\"LabelSelector\":\"\",\"IngressClass\":\"\",\"IngressEndpoint\":null},\"Mesos\":null,\"Eureka\":null,\"ECS\":null,\"Rancher\":null,\"DynamoDB\":null,\"ServiceFabric\":null,\"Rest\":null,\"API\":null,\"Metrics\":null,\"Ping\":null,\"HostResolver\":null}"
time="2018-12-08T05:56:53Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/basics/#collected-data\n"
time="2018-12-08T05:56:53Z" level=debug msg="Add certificate for domains *.my-domain.com"
time="2018-12-08T05:56:53Z" level=debug msg="Creating entry point redirect http -> https"
time="2018-12-08T05:56:53Z" level=info msg="Preparing server http &{Address::80 TLS:<nil> Redirect:0xc0000e5fc0 Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc000418480} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
time="2018-12-08T05:56:53Z" level=info msg="Preparing server https &{Address::443 TLS:0xc0002ed050 Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc000418520} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
time="2018-12-08T05:56:53Z" level=info msg="Starting server on :80"
time="2018-12-08T05:56:53Z" level=debug msg="Add certificate for domains *.my-domain.com"
time="2018-12-08T05:56:53Z" level=info msg="Starting server on :443"
time="2018-12-08T05:56:53Z" level=info msg="Starting provider configuration.ProviderAggregator {}"
time="2018-12-08T05:56:53Z" level=info msg="Starting provider *kubernetes.Provider {\"Watch\":true,\"Filename\":\"\",\"Constraints\":[],\"Trace\":false,\"TemplateVersion\":0,\"DebugLogGeneratedTemplate\":false,\"Endpoint\":\"\",\"Token\":\"\",\"CertAuthFilePath\":\"\",\"DisablePassHostHeaders\":false,\"EnablePassTLSCert\":false,\"Namespaces\":null,\"LabelSelector\":\"\",\"IngressClass\":\"\",\"IngressEndpoint\":null}"
time="2018-12-08T05:56:53Z" level=debug msg="Using Ingress label selector: \"\""
time="2018-12-08T05:56:53Z" level=info msg="ingress label selector is: \"\""
time="2018-12-08T05:56:53Z" level=info msg="Creating in-cluster Provider client"
time="2018-12-08T05:56:53Z" level=debug msg="Received Kubernetes event kind *v1beta1.Ingress"
time="2018-12-08T05:56:53Z" level=error msg="Error configuring TLS for ingress kube-system/traefik-web-ui: secret kube-system/traefik-ui-tls-cert does not exist"
time="2018-12-08T05:56:53Z" level=debug msg="Configuration received from provider kubernetes: {}"
time="2018-12-08T05:56:53Z" level=debug msg="Add certificate for domains *.my-domain.com"
time="2018-12-08T05:56:53Z" level=info msg="Server configuration reloaded on :80"
time="2018-12-08T05:56:53Z" level=info msg="Server configuration reloaded on :443"
time="2018-12-08T05:56:53Z" level=debug msg="Received Kubernetes event kind *v1.Secret"
time="2018-12-08T05:56:53Z" level=warning msg="Endpoints not available for kube-system/traefik-web-ui"
time="2018-12-08T05:56:53Z" level=debug msg="Received Kubernetes event kind *v1.Secret"
time="2018-12-08T05:56:53Z" level=warning msg="Endpoints not available for kube-system/traefik-web-ui"
time="2018-12-08T05:56:53Z" level=debug msg="Skipping Kubernetes event kind *v1.Secret"
time="2018-12-08T05:56:53Z" level=debug msg="Configuration received from provider kubernetes: {\"backends\":{\"traefik-my-machine.my-domain.com/\":{\"loadBalancer\":{\"method\":\"wrr\"}}},\"frontends\":{\"traefik-my-machine.my-domain.com/\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"traefik-my-machine.my-domain.com/\",\"routes\":{\"/\":{\"rule\":\"PathPrefix:/\"},\"traefik-my-machine.my-domain.com\":{\"rule\":\"Host:traefik-my-machine.my-domain.com\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null}}}"
time="2018-12-08T05:56:53Z" level=debug msg="Received Kubernetes event kind *v1.Endpoints"
time="2018-12-08T05:56:53Z" level=debug msg="Received Kubernetes event kind *v1.Endpoints"
time="2018-12-08T05:56:53Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints"
time="2018-12-08T05:56:53Z" level=debug msg="Configuration received from provider kubernetes: {\"backends\":{\"traefik-my-machine.my-domain.com/\":{\"servers\":{\"traefik-ingress-controller-rt5kn\":{\"url\":\"http://10.1.83.5:8080\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}}},\"frontends\":{\"traefik-my-machine.my-domain.com/\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"traefik-my-machine.my-domain.com/\",\"routes\":{\"/\":{\"rule\":\"PathPrefix:/\"},\"traefik-my-machine.my-domain.com\":{\"rule\":\"Host:traefik-my-machine.my-domain.com\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null}}}"
time="2018-12-08T05:56:54Z" level=debug msg="Received Kubernetes event kind *v1.Endpoints"
time="2018-12-08T05:56:54Z" level=debug msg="Received Kubernetes event kind *v1.Endpoints"
time="2018-12-08T05:56:54Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints"
time="2018-12-08T05:56:54Z" level=debug msg="Configuration received from provider kubernetes: {\"backends\":{\"traefik-my-machine.my-domain.com/\":{\"servers\":{\"traefik-ingress-controller-rt5kn\":{\"url\":\"http://10.1.83.5:8080\",\"weight\":1},\"traefik-ingress-controller-tv5lg\":{\"url\":\"http://10.1.77.16:8080\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}}},\"frontends\":{\"traefik-my-machine.my-domain.com/\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"traefik-my-machine.my-domain.com/\",\"routes\":{\"/\":{\"rule\":\"PathPrefix:/\"},\"traefik-my-machine.my-domain.com\":{\"rule\":\"Host:traefik-my-machine.my-domain.com\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null}}}"
time="2018-12-08T05:56:54Z" level=debug msg="Received Kubernetes event kind *v1.Endpoints"
time="2018-12-08T05:56:54Z" level=debug msg="Received Kubernetes event kind *v1.Endpoints"
time="2018-12-08T05:56:54Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints"
time="2018-12-08T05:56:54Z" level=debug msg="Configuration received from provider kubernetes: {\"backends\":{\"traefik-my-machine.my-domain.com/\":{\"servers\":{\"traefik-ingress-controller-bgc5p\":{\"url\":\"http://10.1.31.4:8080\",\"weight\":1},\"traefik-ingress-controller-rt5kn\":{\"url\":\"http://10.1.83.5:8080\",\"weight\":1},\"traefik-ingress-controller-tv5lg\":{\"url\":\"http://10.1.77.16:8080\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}}},\"frontends\":{\"traefik-my-machine.my-domain.com/\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"traefik-my-machine.my-domain.com/\",\"routes\":{\"/\":{\"rule\":\"PathPrefix:/\"},\"traefik-my-machine.my-domain.com\":{\"rule\":\"Host:traefik-my-machine.my-domain.com\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null}}}"
time="2018-12-08T05:56:55Z" level=debug msg="Received Kubernetes event kind *v1.Endpoints"
time="2018-12-08T05:56:55Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints"
time="2018-12-08T05:56:55Z" level=debug msg="Received Kubernetes event kind *v1.Endpoints"
time="2018-12-08T05:56:55Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints"
time="2018-12-08T05:56:55Z" level=debug msg="Add certificate for domains *.my-domain.com"
time="2018-12-08T05:56:55Z" level=debug msg="Wiring frontend traefik-my-machine.my-domain.com/ to entryPoint http"
time="2018-12-08T05:56:55Z" level=debug msg="Creating backend traefik-my-machine.my-domain.com/"
time="2018-12-08T05:56:55Z" level=debug msg="Adding TLSClientHeaders middleware for frontend traefik-my-machine.my-domain.com/"
time="2018-12-08T05:56:55Z" level=debug msg="Creating load-balancer wrr"
time="2018-12-08T05:56:55Z" level=debug msg="Creating server traefik-ingress-controller-bgc5p at http://10.1.31.4:8080 with weight 1"
time="2018-12-08T05:56:55Z" level=debug msg="Creating server traefik-ingress-controller-rt5kn at http://10.1.83.5:8080 with weight 1"
time="2018-12-08T05:56:55Z" level=debug msg="Creating server traefik-ingress-controller-tv5lg at http://10.1.77.16:8080 with weight 1"
time="2018-12-08T05:56:55Z" level=debug msg="Creating route / PathPrefix:/"
time="2018-12-08T05:56:55Z" level=debug msg="Creating route traefik-my-machine.my-domain.com Host:traefik-my-machine.my-domain.com"
time="2018-12-08T05:56:55Z" level=debug msg="Wiring frontend traefik-my-machine.my-domain.com/ to entryPoint https"
time="2018-12-08T05:56:55Z" level=debug msg="Creating backend traefik-my-machine.my-domain.com/"
time="2018-12-08T05:56:55Z" level=debug msg="Adding TLSClientHeaders middleware for frontend traefik-my-machine.my-domain.com/"
time="2018-12-08T05:56:55Z" level=debug msg="Creating load-balancer wrr"
time="2018-12-08T05:56:55Z" level=debug msg="Creating server traefik-ingress-controller-rt5kn at http://10.1.83.5:8080 with weight 1"
time="2018-12-08T05:56:55Z" level=debug msg="Creating server traefik-ingress-controller-tv5lg at http://10.1.77.16:8080 with weight 1"
time="2018-12-08T05:56:55Z" level=debug msg="Creating server traefik-ingress-controller-bgc5p at http://10.1.31.4:8080 with weight 1"
time="2018-12-08T05:56:55Z" level=debug msg="Creating route / PathPrefix:/"
time="2018-12-08T05:56:55Z" level=debug msg="Creating route traefik-my-machine.my-domain.com Host:traefik-my-machine.my-domain.com"
time="2018-12-08T05:56:55Z" level=debug msg="No entryPoint is defined to add the certificate MIIGCzCCBPOgAwIBAgISA0CUiZQjCkQ0GhkaYpjFej8sMA0GCS, it will be added to the default entryPoints: http, https"
time="2018-12-08T05:56:55Z" level=debug msg="Add certificate for domains *.my-domain.com"
time="2018-12-08T05:56:55Z" level=debug msg="Add certificate for domains *.my-domain.com"
time="2018-12-08T05:56:55Z" level=info msg="Server configuration reloaded on :443"
time="2018-12-08T05:56:55Z" level=debug msg="Certificates not added to non-TLS entryPoint http."
time="2018-12-08T05:56:55Z" level=info msg="Server configuration reloaded on :80"
time="2018-12-08T05:56:57Z" level=debug msg="Received Kubernetes event kind *v1.Endpoints"
time="2018-12-08T05:56:57Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints"
time="2018-12-08T05:56:57Z" level=debug msg="Received Kubernetes event kind *v1.Endpoints" |
|
More logs attached... @dtomcej I tried with |
|
I am reviewing these logs, and it appears that the certificate is found, and applied. The certificate is loaded asynchronously with the ingress, so there is a period of time that the secret will log as "not found" before the k8s api provides it. This issue is logged in: #3667. time="2018-12-18T16:38:45Z" level=debug msg="Received Kubernetes event kind *v1beta1.Ingress"
time="2018-12-18T16:38:45Z" level=error msg="Error configuring TLS for ingress kube-system/traefik-web-ui-ingress: secret kube-system/traefik-ui-tls-cert does not exist"
time="2018-12-18T16:38:45Z" level=debug msg="Configuration received from provider kubernetes: {}"
time="2018-12-18T16:38:45Z" level=info msg="Server configuration reloaded on :80"
time="2018-12-18T16:38:45Z" level=info msg="Server configuration reloaded on :443"
time="2018-12-18T16:38:45Z" level=debug msg="Received Kubernetes event kind *v1.Secret"
<--------Secret is now loaded, and no more error log ---------->
time="2018-12-18T16:38:45Z" level=debug msg="Received Kubernetes event kind *v1.Secret"
time="2018-12-18T16:38:45Z" level=debug msg="Skipping Kubernetes event kind *v1.Secret"
<---------If Secret was not loaded, the error message would repeat here ----->
time="2018-12-18T16:38:45Z" level=debug msg="Configuration received from provider kubernetes: {\"backends\":{\"my.server.com\":{\"servers\":{\"traefik-ingress-controller-df74c\":{\"url\":\"http://172.17.0.8:8080\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}}},\"frontends\":{\"my.server.com\":{\"entryPoints\":[\"http\",\"https\"],\"backend\":\"my.server.com\",\"routes\":{\"my.server.com\":{\"rule\":\"Host:my.server.com\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null}}}"The issue that you appear to be reporting is that the http -> https redirect does not work. Can you provide us with the result of a |
dtomcej
changed the title
If your above statement is correct, then the following is a very confusing log message and must be corrected too... Here is the output you asked for. Redirect works, so not sure why you think redirect doesn't work -> curl2.txt |
|
Ah, sorry @aug70, I misread the issue. I thought the redirect was not working also. I read your curl.txt file, and it shows that you connected with TLS 1.2 successfully... The only issue that appears is the 502, but that is a completely different issue altogether. Can you confirm that with curl you are able to connect to traefik on your certificate domains? |
dtomcej
added
priority/P2
|
Yes, this is not a redirect issue. I can connect with curl and I see redirect working. |
aug70
changed the title
|
@dtomcej I'm not sure what are you waiting from me? I have added all the log files that I could possibly add to this issue. This is not a redirect problem! It's a problem with TLS (https endpoint). You should focus on this error message:
Once this is resolved, there won't be any problems (neither redirect nor 502) This seems like a primal functionality for traefik on kubernetes. I'm surprised that it doesn't work and also surprised that you reduced priority to P2. |
|
Hello @aug70 . From your results: s curl -v https://my.server.com/
* Trying 10.225.192.55...
* TCP_NODELAY set
* Connected to my.server.com (10.225.192.55) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=*.server.com
* start date: Dec 8 03:46:35 2018 GMT
* expire date: Mar 8 03:46:35 2019 GMT
* subjectAltName: host "my.server.com" matched cert's "*.server.com"
* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55ea1034e840)At this point, your client is connected via TLS. There are no issues with TLS, or the certificate. > GET / HTTP/2
> Host: my.server.com
> User-Agent: curl/7.58.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!At this point, your client has sent a request to traefik successfully. < HTTP/2 502
< content-type: text/plain; charset=utf-8
< content-length: 11
< date: Fri, 21 Dec 2018 05:23:44 GMT
<
* Connection #0 to host my.server.com left intact
Bad GatewayThis result is that your backend is not responding properly. Traefik cannot connect to the pods on your service defined in your ingress. I would try curling your service and service port that you have defined in your ingress to see if it responds correctly. |
|
Thanks for opening this issue! We need further information to better understand the problem you're facing 🤔 Could you please join us on our Slack workspace and reach out to us on the (#support channel)? We're looking forward to talking to you there! |
|
This is trying to connect to Traefik admin UI that runs on port 8080. |
This comment has been minimized.
This comment has been minimized.
|
@yniknafs Your image shows that the secret Also of note, you have configured traefik to listen on port 443 for TLS via the config, but have not exposed this port as a containerport. I am going to mark your comment as off-topic, so as not to confuse the OP in this issue. |
|
I had the exact same problem. But it was not because of Traefik's configuration. However, even if this is working now, I'm still having this error: Might be related to #4711 ? |
|
Trying to get this working with docker-for-desktop; same problem as above; https://stackoverflow.com/questions/55777815/traefik-not-finding-tls-secret |
|
Hello,
I have tried to see the outout when the secret does not exist for real and the output is completely different, secret-does-not-exist.txt. For the case when the secret exists but the server is not started, secret-not-found.txt. Any ideas are welcome. P.S. I have already tried changing the order of the entry points property as |
|
Similar story here: got the "does not exist" after replacing the certs. I can get/describe the certs with |
|
@jmaerki I wonder if you have this issue resolved? we are facing the same issue where traefik kept serving the old cert. thanks |
SantoDE
removed
the
kind/bug/possible
|
after I did change the sequence/order in the traefik configMap of traefik.toml (having now the [entryPoints.https] section before the [entryPoints.http] section) and doing a I see: note in the middle: my self-signed cert is valid for so it should be also valid for the host: vault.iac.local in my ingress definition: curl'ing the according service directly using the same cert files (via a port-forward: but via ingress: gives me the I don't understand:
|
|
This issue targets Traefik v1, which is not maintained anymore. |
Do you want to request a feature or report a bug?
Bug
What did you do?
Deploy with tls config
What did you expect to see?
Normal operation with http redirecting to https
What did you see instead?
time="2018-11-24T04:57:18Z" level=error msg="Error configuring TLS for ingress kube-system/traefik-web-ui: secret kube-system/traefik-ui-tls-cert does not exist"Output of
traefik version: (What version of Traefik are you using?)What is your environment & configuration (arguments, toml, provider, platform, ...)?
If applicable, please paste the log output in DEBUG level (
--logLevel=DEBUGswitch)When I check the secret it's there...
This has been mentioned in another ticket and closed without much investigation. (I'll reference the ticket here if I can find it)
The text was updated successfully, but these errors were encountered: