Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow immediate renewal of certs if requested. #4539

Closed
SpectralHiss opened this issue Oct 20, 2021 · 6 comments
Closed

Allow immediate renewal of certs if requested. #4539

SpectralHiss opened this issue Oct 20, 2021 · 6 comments
Labels
kind/feature Categorizes issue or PR as related to a new feature. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.

Comments

@SpectralHiss
Copy link
Contributor

This requests stems from a situation where a Venafi TPP issuer went offline temporarily due to a hardware failure.
On the issuance side the failures resulted in a few timeouts temporarily.
As the backend issuer went back online the cert renewal didn't happen immediately.
We suspected this was just due to normal backoff on retries so a kubectl cert-manager renew was attempted to immediately renew.

To our surprise this still didn't renew the certificate.

Looking at the cert-manager controller logs we saw messages like so:
cert-manager/controller/certificates-trigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="istio-system/test-cert" "retry_delay"=3599178696277

Therefore we assumed from that log that the rate limit avoidance designed to protect ACME servers blocked us.
This is a good default that perhaps we'd like to be able to override in some cases.

This wasn't crucial as waiting 1 hour would solve the issue, in most cases one can wait without much issue.
It would make for a nicer UX and ease of operations if one has the option to force an immediate renewal.

Describe the solution you'd like

If there is a way to mark the certificate for immediate renewal (ignoring the rate limiting prevention measure) , we'd like to be able to use it.

@JoshVanL suggested adding a --force flag on the kubectl cert-manager renew command for this use case which i think is an adequate solution.

Another more complicated, nice to have general alternative is to have the rate limiting behaviour somehow controlled in the issuer, it would depend on the various backend issuers, for example Vault allows custom api quotas configuration, which we could have a similar config for.
We'd probably still want the force flag anyways in some edge cases even with above configuration.

Environment details:

  • Kubernetes version:
    GKE 1.20.9
  • cert-manager version:
    1.5.4

/kind feature

@jetstack-bot jetstack-bot added the kind/feature Categorizes issue or PR as related to a new feature. label Oct 20, 2021
@munnerz
Copy link
Member

munnerz commented Nov 17, 2021

Hm, the way that the cmctl renew command works is that it bypasses the trigger controller by directly adding the Issuing condition to the Certificate resource.

Are you sure that this was actually the most up to date reconciliation of the certificate controllers? That error message shouldn't be an issue when using cmctl renew.

@SpectralHiss
Copy link
Contributor Author

ah interesting, at the time i was using the regular plugin v1.5.4 (so using kubectl cert-manager renew cert ) has the logic changed since then?

For some people they like not even have to use the cli at all in which case having a per issuer rate limiting config would be nice but this was not really a high-priority issue at all, just a nice-to-have.

@irbekrm
Copy link
Collaborator

irbekrm commented Dec 3, 2021

It is possible that it was this issue #4642

If the bug linked above would have been hit then the first renewal would have resulted in successful CertificateRequests, but only the second renewal (i.e after 1 hour backoff) would have resulted in the new TLS certs synced to Secrets and the Certificates statuses updated.

@jetstack-bot
Copy link
Collaborator

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

@jetstack-bot jetstack-bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 3, 2022
@jetstack-bot
Copy link
Collaborator

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle rotten
/remove-lifecycle stale

@jetstack-bot jetstack-bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Apr 2, 2022
@irbekrm
Copy link
Collaborator

irbekrm commented Apr 28, 2022

cmctl renew does renew immediately (without the 1 hour backoff)

I think the issue here may have been #4642 that was fixed in cert-manager 1.7

Do let us know if that is still an issue.

@irbekrm irbekrm closed this as completed Apr 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Categorizes issue or PR as related to a new feature. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.
Projects
None yet
Development

No branches or pull requests

4 participants