Describe the bug:
The acmesolver pods for some challenges are created and removed in a loop, and cert-manager can't make progress.
multiple existing challenge solver pods found and cleaned up. retrying challenge sync
redacted-logs.txt
Expected behaviour:
- Somehow, only one pod to be created per challenge.
- When it so happens that there are multiple pods created per challenge - the cert manager should not get stuck in a loop of creating and removing them, and should, instead be able to proceed - like, remove all pods but one, or sth.
Steps to reproduce the bug:
Idk what happened that caused this bug.
Anything else we need to know?:
I suspect it started with k8s somehow creating more than one pod in the first place (i.e. due to eviction) and cert-manager got two error notifications via the watch subscription, handled them each be re-creating the pods - and that's how the loops starts; but this is speculation.
It was working for ages until it broke down ~22 days ago, we were at version v1.13.x. Today I've updated to v1.17.2 - it didn;t solve the issue. We didn't change anything in the setup.
Out of 200+ ingresses with certs, only ~6 are at this state. 2 new ones, and ~4-ish renewing - same odd behavior. Different namespaces, but same ingress controllers and ClusterIssuer. Using http01 solver (you know, the one that needs to create pods).
Switching to a different ClusterIssuer that has a dns01 instead of http01 makes the ingress get a certificate quickly and correctly, however, when switched back to the http01-powered ClusterIssue, the certificate renew is attempted and the issue returns.
I'm hesitant to experiment on this stuff more - as there is a potential risk of losing certificates of all our ingresses.
Environment details:
-
Kubernetes version:
Client Version: v1.32.3
Kustomize Version: v5.5.0
Server Version: v1.32.3
-
Cloud-provider/provisioner: scaleway
-
cert-manager version: v1.17.2
-
Install method: helm
/kind bug
Describe the bug:
The
acmesolverpods for some challenges are created and removed in a loop, and cert-manager can't make progress.multiple existing challenge solver pods found and cleaned up. retrying challenge syncredacted-logs.txt
Expected behaviour:
Steps to reproduce the bug:
Idk what happened that caused this bug.
Anything else we need to know?:
I suspect it started with k8s somehow creating more than one pod in the first place (i.e. due to eviction) and cert-manager got two error notifications via the
watchsubscription, handled them each be re-creating the pods - and that's how the loops starts; but this is speculation.It was working for ages until it broke down ~22 days ago, we were at version v1.13.x. Today I've updated to v1.17.2 - it didn;t solve the issue. We didn't change anything in the setup.
Out of 200+ ingresses with certs, only ~6 are at this state. 2 new ones, and ~4-ish renewing - same odd behavior. Different namespaces, but same ingress controllers and
ClusterIssuer. Usinghttp01solver (you know, the one that needs to create pods).Switching to a different
ClusterIssuerthat has adns01instead ofhttp01makes the ingress get a certificate quickly and correctly, however, when switched back to thehttp01-poweredClusterIssue, the certificate renew is attempted and the issue returns.I'm hesitant to experiment on this stuff more - as there is a potential risk of losing certificates of all our ingresses.
Environment details:
Kubernetes version:
Client Version: v1.32.3
Kustomize Version: v5.5.0
Server Version: v1.32.3
Cloud-provider/provisioner: scaleway
cert-manager version: v1.17.2
Install method: helm
/kind bug