TLDR;
cert-manager kills the challenge + resolver pod/ingress before letsencrypt secondary validation finishes.
The hack with removing the 'delete/challenges' permission on the 'cert-manager-controller-orders' role and 'delete/(pods+ingresses)' on the 'cert-manager-controller-challenges' role until AFTER the secondary validation is finished and letsencrypt is finally satisfied - IT WORKS (found here: #4676 (comment)).
My setup:
- RKE2-based self hosted 3-node kubernetes cluster (currently v1.31.5+rke2r1).
- one reverse proxy with global fixed IP assigned explicitly from ISP routes all 80/443 traffic to one fixed node in the cluster.
- one internet exposed DNS resolving my domain and also the hosts from that domain explicitly by name to the above gloabl-fixed-IP reverse proxy
- above DNS has A and CAA records (letsencrypt) for the two domains and A records for each of the hosts
- cert-manager 'helm'-installed (currently v1.17.1) in the RKE2-cluster
- two cluster issuers configured (staging and prod)
- have ingresses for all corresponding 'clustered'-hosts, annotated to use one of the cluster issuers, configured for http-01 challenges
- letsdebug.net indicates 'All OK!' for http-01 when run against any of the ingresses/hosts in the cluster
This setup has been working wonderfully for almost a year, untill about a month ago (2025/03) when certificate renewal requests started becoming unavailable with orders either in 'pending' or 'invalid' states. Some of the certificate requests would eventually run through completion after a (very long) while, but most do not within the time of validity left.
Digging through logs and trying manually triggered certificate renewals I have come to the conclusion that the challenges are resolved correctly, however, by the time letsencrypt performs the secondary validation the challenge and corresponding resolver resources are no longer available, cert-manager kills the pods.
Question: isn't there any other way than the mentioned hack (#4676 (comment)) to allow the challenge&co to 'live' a little longer? I mean I have 50+ (and growing) domains in this cluster and managing the certificates' renewal this way will get... well... 'painful'.
Alternately I would find it tremendously useful if someone could help me understand what is triggering the kill of the challenge&co sooner than what would allow the letsencrypt validation flow to run thru' completion. And eventually what can I do about it.
Many thanks.
TLDR;
cert-manager kills the challenge + resolver pod/ingress before letsencrypt secondary validation finishes.
The hack with removing the 'delete/challenges' permission on the 'cert-manager-controller-orders' role and 'delete/(pods+ingresses)' on the 'cert-manager-controller-challenges' role until AFTER the secondary validation is finished and letsencrypt is finally satisfied - IT WORKS (found here: #4676 (comment)).
My setup:
This setup has been working wonderfully for almost a year, untill about a month ago (2025/03) when certificate renewal requests started becoming unavailable with orders either in 'pending' or 'invalid' states. Some of the certificate requests would eventually run through completion after a (very long) while, but most do not within the time of validity left.
Digging through logs and trying manually triggered certificate renewals I have come to the conclusion that the challenges are resolved correctly, however, by the time letsencrypt performs the secondary validation the challenge and corresponding resolver resources are no longer available, cert-manager kills the pods.
Question: isn't there any other way than the mentioned hack (#4676 (comment)) to allow the challenge&co to 'live' a little longer? I mean I have 50+ (and growing) domains in this cluster and managing the certificates' renewal this way will get... well... 'painful'.
Alternately I would find it tremendously useful if someone could help me understand what is triggering the kill of the challenge&co sooner than what would allow the letsencrypt validation flow to run thru' completion. And eventually what can I do about it.
Many thanks.