Exponential backoff on CA failures

[howardjohn]

Currently here we retry after 60s. 60s can be a long time or a short time. We should probably use a backoff scheme. There may be transient issues in the CA (e.g. the SA informer cache is a bit stale) that will resolve after 1s.. but if its never going to approve, retrying 1440 times a day may be a bit much.

[keithmattix]

/cc @MorrisLaw

[MorrisLaw]

Thank you Keith, I'll look into this!

[MorrisLaw]

Is there a retry library that maintainers recommend @howardjohn?

[hzxuzhonghu]

It is not limited, but looks https://crates.io/crates/backoff is the most popular one

[MorrisLaw]

PR is ready for review again: #768

[MorrisLaw]

PR has been merged.

[keithmattix]

Had to revert; reopening

[MorrisLaw]

Had to revert due to an issue noticed in an automated Istio PR that tried updating to the latest stable ztunnel master. TestIntermediateCertificateRefresh failed due to missing cert. It looks like it wasn't refreshed in time or it could be a possible race condition.

Related areas of the code to look at as it relates to the integration test that were changed before and are relevant to the backoff PR, these may require modification in order to support this change:

• https://github.com/istio/ztunnel/pull/742/files
• istio/istio@1ee1a43

Goal is to reproduce the integration test failure with the original PR that has been reverted, then figure out how to update the integration test or the business logic itself so that the integration test no longer fails.

[MorrisLaw]

This has been merged back in. Found the issue in an integration test and fixed it. Also, confirmed that istio integration is happy with this latest version of ztunnel: https://github.com/istio/istio/pull/50079/files. We can close this for real now.