SDS falis to load available secret #18061
Comments
|
OK, I found the problem and it is a bug. The issue is that cert-manager creates secrets with 3 keys, like So I first tried deleting the ca.crt key from the secret and cert-manager just adds it right back, so I then edited the secret's ca.crt key to have the base64 encoded value of 'foo' which is 'Zm9v' and updated the secret. Once I did this then I got the following in the ingress-sds logs And executiong echo | openssl s_client -showcerts -servername httpbin.foo.io -connect httpbin.foo.io:443 2>/dev/null | openssl x509 -inform pem -noout -text shows the updated cert's info. Thus the ingress-sds code needs to be updated to either ignore zero length values for keys it doesn't care about or only check the values of keys it does care about (i.e. tls.key and tls.crt). |
|
Looks like this should stop happening with cert-manager v0.11.0 because they will no longer be generating a temporary certificate while waiting for the ACME cert to be issued: https://github.com/jetstack/cert-manager/releases/tag/v0.11.0 |
|
@pib I'm already using cert-manager 0.11. |
|
I also see the described behaviour. I was able to have SDS load my keypair generated by certmanager/letsencrypt only once and still trying to reproduce that with no luck. At the same time I was not able to completely reproduce the trick with manually replacing Another observation:
Here is the same logline with stack trace enabled: and here is the brand new secret described after it was regenerated from scratch: |
|
Ok, I think I identified the failing scenario. @wpbeckwith did you create gateway in the same yaml manifest as your gateway? (I assume you tried to attach it to the gateway as that's what I was doing). If I create certificate and attach it to the gateway HTTPS endpoint in a single Here is related helm template part I use (redacted) for reference. |
|
We faced the same issue when migrating from istio 1.2 to 1.3 (confirmed with 1.3.3). We're using cert-manager to generate our certicate through Let's Encrypt. Istio Gateway is created at the same time we ask for the certificate, so before the certificate challenge being accepted. Once secret is generated/updated: Unfortunately the certificate used is issued by If I create a secret with valid certificate, then create gateway asking sds to use that secret it works fine. Then if I edit secret and replace with a certificate for another domain, initial certificate is still used and requests succeed. It definitely looks like sds is unable to take secret update into account anymore. |
|
I wonder what happens when Letsencrypt certificate expires and needs to be renewed. |
|
This seems to be fixed in 1.3.4 |
|
I'll checkout 1.3.4 this weekend and report back. |
|
I've just checked this issue with 1.3.4 and I don't confirm this is fixed. |
|
I could not confirm it is fixed in 1.4.0 as well |
|
@timurb Could you describe how you test it with Istio 1.4.0, and provide node agent logs so that I can take a look? |
|
Sure.
gateways:
enabled: true
istio-ingressgateway:
enabled: true
sds:
enabled: true
sidecarInjectorWebhook:
enabled: true
security:
enabled: true
certmanager:
enabled: false
global:
tag: 1.4.0
mtls:
enabled: false
sds:
enabled: false
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: release-name-istio-gateway
namespace: kafka-services
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http-80-release-name-istio-gateway
protocol: HTTP
hosts:
- "*"
- port:
number: 443
name: https-443-release-name-istio-gateway
protocol: HTTPS
tls:
mode: SIMPLE # enables HTTPS on this port
credentialName: release-name-istio-gateway
privateKey: sds
serverCertificate: sds
hosts:
- "*"
When certificate turns into READY state according to output of Alternatively I can make it work perfectly if I don't specify SSL port to ingressgateway definition at the very beginning but instead edit and add it once certificate gets to READY state. In that case running Logs of ingressgateway when I see the problem I'm not completely confident that the loglines were the same for istio 1.3.x, please let me know if you want me to verify that. Logs of everything working fine when I don't specify HTTPS endpoint for ingressgateway and create it later when secret is ready. |
|
@timurb Thanks for the update. Is it possible for you to dump the secret istio-kafkagateway-istio-gateway, and check if the secret has proper set of fields? {"cert", "key"} or {"tls.crt", "tls.key"}One more thing is, please don't use prefix "istio" for gateway secrets, as we reserve that prefix for istio internal secrets. This secret should be filtered out here but it gets parsed somehow. |
Sorry for misleading. I'm using correct name for secret -- the one that shows up in logs.
Right, this is correct log message: while certificate is not yet issued by letsencrypt one of the fields are empty. Once cert-manager passes challenge-response successfully with letsencrypt it updates the secret with certificate and private key.
Thanks for reminding about that! |
|
@JimmyCYJ looks like that was my problem — having I checked that only in istio 1.4.0. If you want me to check that in 1.3.X please let me know. |
|
Thanks for letting me know. I am going to close this issue now. Please feel free to reopen it. |
Bug description
I have installed istio with SDS enabled and all works fine except that secrets are not getting reloaded by SDS. I can successfully submit a cert-manager certificate like
And the secret will be created in the istio-system namesapce.
However the logs for the ingress-sds process show
The above logs show where I deleted the certificate and the secret from the istio-system namespace and then reapplied the above certificate. If i execute echo | openssl s_client -showcerts -servername httpbin.foo.io -connect httpbin.foo.io:443 2>/dev/null | openssl x509 -inform pem -noout -text then I can see that the cert returned is the old cert and not the new. So far, killing the pod is the only way to get the new cert loaded.
Affected product area (please put an X in all that apply)
[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[X ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Expected behavior
The secret should be reloaded when changed by cert-manager.
Steps to reproduce the bug
Version (include the output of
istioctl version --remoteandkubectl version)istioctl version --remote
client version: 1.3.2
ingressgateway version: 1.3.2
pilot version: 1.3.2
kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"clean", BuildDate:"2019-06-07T09:55:27Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.7-eks-e9b1d0", GitCommit:"e9b1d0551216e1e8ace5ee4ca50161df34325ec2", GitTreeState:"clean", BuildDate:"2019-09-21T08:33:01Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
How was Istio installed?
With helm
Environment where bug was observed (cloud vendor, OS, etc)
AWS EKS
Additionally, please consider attaching a cluster state archive by attaching
the dump file to this issue.
The text was updated successfully, but these errors were encountered: