istiod failed to renew certificates in sidecars and gateways

[uliromahn]

Bug Description

This is kind of an edge case but I believe still worth reporting.

I installed Istio into my kind cluster using the demo profile which installs both ingress- as well as the egress gateway.
I was configuring Istio to not allow access to external services and then created a ServiceEntry and corresponding VirtualServices and DestinationRules to access the external service from my service pod.
I also configured Istio to use mTLS between the pod and my egress-gateway.

Everything worked fine until I "shut down" my Kind cluster. However, instead of deleting my cluster, I stopped the Kind container using docker stop kind-control-node.

I then left my cluster for a couple of days after which I woke it up using docker start ....

However, all in a sudden my service pod was unable to reach my external service. Enabling DEBUG mode on Envoy in the egress gateway gave me a connection error:
2022-03-22T23:56:16.247595Z debug envoy connection [C11395] TLS error: 268436501:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_EXPIRED
Getting the certs from the Envoy admin interface showed me that several certificates had actually expired. However, the days to expire showed a number greater than zero. Because of this, istiod was not renewing the certificate.
It appears that the value holding the days to expire is an unsigned long (or int) and because istiod was not running at the time of expiration, it is now unable to detect the expiration.

I suspect that this could also be a bug in Envoy Proxy. Please let me know if I should submit a separate bug report there or if someone from the Istio community will do that.

Version

Istio 1.13.1

Additional Information

No output since all necessary information should be contained above.

[uliromahn]

After some additional investigation, it appears that this is a clear bug in Envoy and not Istio. I will hence close this issue here and report it in the Envoy git repo.

[ramaraochavali]

@uliromahn For posterity, would you mind sharing the envoy bug/issue?