how often does istio-proxy push the sds

[yalctay93]

We are using the credentialName field of kind gateway in our istio setup in order to mount server certifictes as well as ca-certificates.

How often does Istio update those secrets in its configuration? Lets say the -cacerts secret has changed, is it updated immediately or do we have to wait for a sds push or something? We use the same as filemount on kind destinationRule as the credentialName feature is not enabled and there it usually takes 10-15 seconds for a secret to be mounted and for changes to be updated aswell.

I've noticed that sometimes the SDS does not work properly in istio version 1.9.5 or maybe I am doing something wrong. Can someone clear me up on this? The SDS push happens in a certain interval or immediately when a secret content such as -cacerts are changed/updated?

Istio logs show me this only:

{"level":"info","time":"2021-06-15T09:28:37.014619Z","scope":"sds","msg":"SDS server for workload certificates started, listening on \"./etc/istio/proxy/SDS\""}
{"level":"info","time":"2021-06-15T09:28:37.015529Z","scope":"sds","msg":"Start SDS grpc server"}
{"level":"info","time":"2021-06-15T09:28:40.111528Z","scope":"sds","msg":"SDS: PUSH","resource":"file-root:/etc/istio/ingressgateway-client-ca-certs-corpintra/ca.crt"}
{"level":"info","time":"2021-06-15T09:28:40.112139Z","scope":"sds","msg":"SDS: PUSH","resource":"file-cert:/etc/istio/ingressgateway-client-acme-certs-mutual/tls.crt~/etc/istio/ingressgateway-client-acme-certs-mutual/tls.key"}
{"level":"info","time":"2021-06-15T09:28:40.214305Z","scope":"sds","msg":"SDS: PUSH","resource":"default"}
{"level":"info","time":"2021-06-15T09:28:40.216426Z","scope":"sds","msg":"SDS: PUSH","resource":"ROOTCA"}
{"level":"info","time":"2021-06-15T09:32:35.829781Z","scope":"sds","msg":"SDS: PUSH","resource":"file-root:/etc/istio/ingressgateway-client-ca-certs-corpintra/ca.crt"}
{"level":"info","time":"2021-06-15T21:28:42.618498Z","scope":"sds","msg":"SDS: PUSH","resource":"default"}

So for today (2021-06-16) I dont have any log entries yet regarding any SDS PUSH however, I have already updated the secrets that are specified by the credentialName field of our gateway a few seconds ago. Shouldn't I expect an SDS push here immediately afterwards?

Furthermore, is there any way to check which secrets were mounted or which secret is currently used that has been mounted through the gateway credentialName field?

Thanks alot!

[ ] Docs
[x] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[x] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
[ ] Upgrade

How was Istio installed?
Istio Operator

[howardjohn]

It's pushed from Istiod, not the istio-proxy (which is only the workload certificate). It should be ~instant You can view loaded secrets with istioctl pc s

[yalctay93]

Much appreciated, that command will be handy for us. Thanks!

[yalctay93]

What I have noticed is, the secret that was specified by gateway credentialName field had changed (we changed the key value from "tls.crt" to "cert") however, that change was never updated in the ingress-gateway.

Example:
Previous secret used by credentialName:

apiVersion: v1
data:
  tls.crt: c3VycHJpc2Ugc3VycHJpc2UsIEkgYW0gYSB0bHMuY3J0Cg==
kind: Secret
metadata:
  name: test-secret
  namespace: istio-system
type: Opaque

Then we updated the secret essentially changing tls.crt to cert:

apiVersion: v1
data:
  cert: c3VycHJpc2Ugc3VycHJpc2UsIEkgYW0gYSB0bHMuY3J0Cg==
kind: Secret
metadata:
  name: test-secret
  namespace: istio-system
type: Opaque

Instead of using the new "cert" file it was still using the tls.crt which was not available anymore and did not update the new secret. I believe it mounted the "tls.crt" by SDS push but never bothered to change it up and replace it with "cert".

Only after I deleted all gateway objects using that specific secret under credentialName field and then re-creating that gateway again, I was able to use the correct secret.

Is this an expected behaviour or a bug?

[howardjohn]

Might be a bug where changing the key doesn't update it. Does it work if you update the value instead of the key?

[yalctay93]

We never had issues with changing the value. Then again, the key is not changed often during production we only changed it up today because I wanted to fit the istio documentation. I dont know if you want to bother and fix this "bug".

[howardjohn]

I could not reproduce this. One thing I noticed - you have only tls.crt, you need tls.key as well. So maybe its storing an old valid version, and you are just flip-flopping between 2 invalid configs which are ignored

[yalctay93]

I could not reproduce this. One thing I noticed - you have only tls.crt, you need tls.key as well. So maybe its storing an old valid version, and you are just flip-flopping between 2 invalid configs which are ignored

I left out the tls.key knowinlgy, sorry if that caused confusion.

[howardjohn]

If you left out tls.key it is not expected to work

…

On Wed, Jun 23, 2021 at 12:01 AM yalctay93 ***@***.***> wrote: I could not reproduce this. One thing I noticed - you have only tls.crt, you need tls.key as well. So maybe its storing an old valid version, and you are just flip-flopping between 2 invalid configs which are ignored I left out the tls.key knowinlgy, sorry if that caused confusion. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#33464 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXKDZUOLIGEXMZI7BKTTUGBEFANCNFSM46Y2WRWA> .

[yalctay93]

If you left out tls.key it is not expected to work
…
On Wed, Jun 23, 2021 at 12:01 AM yalctay93 @.***> wrote: I could not reproduce this. One thing I noticed - you have only tls.crt, you need tls.key as well. So maybe its storing an old valid version, and you are just flip-flopping between 2 invalid configs which are ignored I left out the tls.key knowinlgy, sorry if that caused confusion. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#33464 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEYGXKDZUOLIGEXMZI7BKTTUGBEFANCNFSM46Y2WRWA .

Sorry if I was not clear enough, by left out knowingly I meant that I did not include it in this post or rather in the above example. In the cluster itself however, the key was obviously present.

[istio-policy-bot]

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2021-06-23. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.