Cert-manager created multiple CertificateRequests (over 30k) for a valid certificate

[sneha1013]

Describe the bug:
Multiple CertificateRequests (over 30k) got created for a certificate in a day even though the certificate has a long validity.

status:                                          
  conditions:                                    
  - lastTransitionTime: "2025-02-07T23:46:17Z"   
  message: Certificate is up to date and has not expired   
  observedGeneration: 1                        
  reason: Ready                                
  status: "True"                               
  type: Ready                                  
  notAfter: "2025-07-30T09:25:42Z"               
  notBefore: "2025-05-31T09:25:42Z"              
  renewalTime: "2025-07-25T09:25:42Z"            
  revision: 30464

Expected behaviour:
The cert-manager should not have created so many CertificateRequests when the certificate is valid.

Steps to reproduce the bug:
We are not sure what caused this issue and were able to reproduce it again.

/kind bug

[api-py]

Please provide the manifest for the ClusterIssuer/Issuer - and remove any secrets, if they exist in the manifest.

As well the Certificate manifest.

Thanks

[sneha1013]

This issue has now happened a few times in different environments. I am providing the manifest for the latest one.
Issuer CR:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  annotations:
    helm.sh/resource-policy: keep
    lcm.private.gdc.goog/claim-by-force: "true"
    meta.helm.sh/release-name: log-audit-backend
    meta.helm.sh/release-namespace: obs-system
  creationTimestamp: "2025-05-05T20:03:06Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
    lcm.private.gdc.goog/component: log
    lcm.private.gdc.goog/subcomponent: log-audit
  name: observability-tls-cert-issuer
  namespace: istio-system
  resourceVersion: "58982"
  uid: 5b964284-bd53-4b37-9a1a-b5e18a27e44e
spec:
  ca:
    secretName: observability-tls-ca
status:
  conditions:
  - lastTransitionTime: "2025-05-05T20:03:11Z"
    message: Signing CA verified
    observedGeneration: 1
    reason: KeyPairVerified
    status: "True"
    type: Ready

Certificate CR:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  creationTimestamp: "2025-08-01T21:48:57Z"
  generation: 1
  labels:
    logging.private.gdch.goog/auditloggingtarget-name: project-grafana-audit-logging-target
    logging.private.gdch.goog/auditloggingtarget-namespace: mon-poet-hvjndgji-obs-system
  name: grafana-440e06b2c574f99604cb5012093096fb-alog-infra-client-tls
  namespace: istio-system
  resourceVersion: "325406992"
  uid: e85c52cc-e177-4076-a498-55104054d476
spec:
  dnsNames:
  - '*'
  duration: 8760h0m0s
  issuerRef:
    kind: Issuer
    name: observability-tls-cert-issuer
  privateKey:
    algorithm: RSA
    rotationPolicy: Always
    size: 4096
  renewBefore: 730h0m0s
  secretName: grafana-440e06b2c574f99604cb5012093096fb-alog-infra-client-tls
  subject:
    countries:
    - US
    localities:
    - Mountain View
    organizationalUnits:
    - GDCH
    organizations:
    - Google LLC
    provinces:
    - CA
  usages:
  - client auth
status:
  conditions:
  - lastTransitionTime: "2025-08-04T18:22:11Z"
    message: The certificate has been successfully issued
    observedGeneration: 1
    reason: Issued
    status: "False"
    type: Issuing
  - lastTransitionTime: "2025-08-04T18:22:11Z"
    message: Certificate is up to date and has not expired
    observedGeneration: 1
    reason: Ready
    status: "True"
    type: Ready
  notAfter: "2026-08-04T18:21:38Z"
  notBefore: "2025-08-04T18:21:38Z"
  renewalTime: "2026-07-05T08:21:38Z"
  revision: 33088

There is another certificate with the same issuer and it shows the message Issuing certificate as Secret does not exist in status conditions. But upon checking we see that the secret does exist.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  creationTimestamp: "2025-08-01T21:47:54Z"
  generation: 1
  labels:
    logging.private.gdch.goog/auditloggingtarget-name: ods-audit-target
    logging.private.gdch.goog/auditloggingtarget-namespace: g-dbs-haas-system-haas-haa-c3ebnzl473kh-sie
  name: dbs-haas-system-haas-haas-3832-aqos4o3pxc9v-alog-infra-client-tls
  namespace: istio-system
  resourceVersion: "325406873"
  uid: 8d307c2d-c6da-4975-8538-00ea7fb58eb1
spec:
  dnsNames:
  - '*'
  duration: 8760h0m0s
  issuerRef:
    kind: Issuer
    name: observability-tls-cert-issuer
  privateKey:
    algorithm: RSA
    rotationPolicy: Always
    size: 4096
  renewBefore: 730h0m0s
  secretName: dbs-haas-system-haas-haas-3832-aqos4o3pxc9v-alog-infra-client-tls
  subject:
    countries:
    - US
    localities:
    - Mountain View
    organizationalUnits:
    - GDCH
    organizations:
    - Google LLC
    provinces:
    - CA
  usages:
  - client auth
status:
  conditions:
  - lastTransitionTime: "2025-08-04T18:22:11Z"
    message: Certificate is up to date and has not expired
    observedGeneration: 1
    reason: Ready
    status: "True"
    type: Ready
  - lastTransitionTime: "2025-08-04T18:21:38Z"
    message: Issuing certificate as Secret does not exist
    observedGeneration: 1
    reason: DoesNotExist
    status: "True"
    type: Issuing
  nextPrivateKeySecretName: dbs-haas-system-haas-haas-3832-aqos4o3pxc9v-alog-infra-cligbpb2
  notAfter: "2026-08-04T18:21:36Z"
  notBefore: "2025-08-04T18:21:36Z"
  renewalTime: "2026-07-05T08:21:36Z"
  revision: 34145

Secret exists:

NAME                                                                TYPE                DATA   AGE
dbs-haas-system-haas-haas-3832-aqos4o3pxc9v-alog-infra-client-tls   kubernetes.io/tls   3      4d13h

[erikgb]

@sneha1013 Which version of cert-manager are you running? And how did you install it? If using Helm, please provide your Helm values (redacting sensitive values).

[sneha1013]

We are using cert-manager version 1.14.7 and installing it using heml charts.
Deployment CR:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "5"
  creationTimestamp: "2025-05-05T19:06:35Z"
  generation: 5
  labels:
    app: cert-manager
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: cert-manager
    app.kubernetes.io/name: cert-manager
    app.kubernetes.io/version: v1.13.2
  name: cert-manager
  namespace: cert-manager
  resourceVersion: "301681122"
  uid: 8cc11c87-7d7a-4cbd-8409-b5abdbe19997
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: cert-manager
      app.kubernetes.io/name: cert-manager
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        monitoring.gke.io/path: /metrics
        monitoring.gke.io/port: "9402"
        monitoring.gke.io/scheme: http
        monitoring.gke.io/scrape: "true"
      creationTimestamp: null
      labels:
        app: cert-manager
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: cert-manager
        app.kubernetes.io/name: cert-manager
        app.kubernetes.io/version: v1.13.2
        egress.networking.gke.io/enabled: "true"
    spec:
      containers:
      - args:
        - --v=2
        - --cluster-resource-namespace=$(POD_NAMESPACE)
        - --leader-election-namespace=kube-system
        - --acme-http01-solver-image=gcr.io/private-cloud-staging/cert-manager-acmesolver:v1.14.7-gke.11
        - --max-concurrent-challenges=60
        - --feature-gates=StableCertificateRequestName=true,UseCertificateRequestBasicConstraints=true,ServerSideApply=true
        - --dns01-recursive-nameservers-only=true
        - --dns01-check-retry-period=60s
        - --kube-api-burst=10000
        - --kube-api-qps=10000
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: gcr.io/private-cloud-staging/cert-manager-controller:v1.14.7-gke.11
        imagePullPolicy: IfNotPresent
        name: cert-manager-controller
        ports:
        - containerPort: 9402
          name: http-metrics
          protocol: TCP
        - containerPort: 9403
          name: http-healthz
          protocol: TCP
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      enableServiceLinks: false
      nodeSelector:
        kubernetes.io/os: linux
        node-role.kubernetes.io/control-plane: ""
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        runAsGroup: 2020
        runAsNonRoot: true
        runAsUser: 2020
        seccompProfile:
          type: RuntimeDefault
      serviceAccount: cert-manager
      serviceAccountName: cert-manager
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2025-05-05T19:06:40Z"
    lastUpdateTime: "2025-05-05T19:06:40Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2025-05-05T19:06:35Z"
    lastUpdateTime: "2025-07-29T02:15:49Z"
    message: ReplicaSet "cert-manager-985ff6bcb" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 5
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

[erikgb]

cert-manager 1.14 is EOL for long. Please upgrade: https://cert-manager.io/docs/releases/#currently-supported-releases

[sneha1013]

We are working on upgrading the version, but meanwhile we would appreciate some help to understand the root cause of this issue.

[jaingeet]

@erikgb for one of the certificate CR, we are seeing this status conditions -

status:
  conditions:
  - lastTransitionTime: "2025-08-04T18:22:11Z"
    message: Certificate is up to date and has not expired
    observedGeneration: 1
    reason: Ready
    status: "True"
    type: Ready
  - lastTransitionTime: "2025-08-04T18:21:38Z"
    message: Issuing certificate as Secret does not exist
    observedGeneration: 1
    reason: DoesNotExist
    status: "True"
    type: Issuing
  nextPrivateKeySecretName: dbs-haas-system-haas-haas-3832-aqos4o3pxc9v-alog-infra-cligbpb2
  notAfter: "2026-08-04T18:21:36Z"
  notBefore: "2025-08-04T18:21:36Z"
  renewalTime: "2026-07-05T08:21:36Z"
  revision: 34145

kind: Secret
metadata:
  annotations:
    cert-manager.io/alt-names: '*'
    cert-manager.io/certificate-name: dbs-haas-system-haas-haas-3832-aqos4o3pxc9v-alog-infra-client-tls
    cert-manager.io/common-name: ""
    cert-manager.io/ip-sans: ""
    cert-manager.io/issuer-group: ""
    cert-manager.io/issuer-kind: Issuer
    cert-manager.io/issuer-name: observability-tls-cert-issuer
    cert-manager.io/subject-countries: US
    cert-manager.io/uri-sans: ""
  creationTimestamp: "2025-08-02T04:13:44Z"
  labels:
    controller.cert-manager.io/fao: "true"
  name: dbs-haas-system-haas-haas-3832-aqos4o3pxc9v-alog-infra-client-tls
  namespace: istio-system
  resourceVersion: "325405406"
  uid: 647dc6de-7ce8-4c1a-948d-1e8ce2b21566
type: kubernetes.io/tls 

Here the lastTransitionTime for condition type issuing is before Ready, so ideally condition type Issuing should be set as False as the CR is marked as ready as per https://cert-manager.io/docs/usage/certificate/#inner-workings-diagram-for-developers. And we are continuously seeing this error in controller logs for this Certificate CR -

E0806 20:14:58.626684       1 controller.go:167] "re-queuing item due to error processing" err="certificaterequests.cert-manager.io \"dbs-haas-system-haas-haas-3832-aqos4o3pxc9v-alog-infra-client-tls-34146\" already exists" logger="cert-manager.certificates-request-manager" key="istio-system/dbs-haas-system-haas-haas-3832-aqos4o3pxc9v-alog-infra-client-tls"       E0806 20:15:29.079727       1 controller.go:167] "re-queuing item due to error processing" err="certificaterequests.cert-manager.io \"dbs-haas-system-haas-haas-3832-aqos4o3pxc9v-alog-infra-client-tls-34146\" already exists" logger="cert-manager.certificates-request-manager" key="istio-system/dbs-haas-system-haas-haas-3832-aqos4o3pxc9v-alog-infra-client-tls"

[jaingeet]

@erikgb if you see secret does exist with a creationTimestamp as "2025-08-02T04:13:44Z" whereas the certificate condition complaining about secret does not exist is at "2025-08-04T18:21:38Z". Do you think it might be facing this caching issue as described here #6494, though I can see that secret does have controller.cert-manager.io/fao: "true" being set.

[erikgb]

There could be a number of causes for this issue, but I don't think anyone is going to spend time trying to get to the bottom of this in an obsolete cert-manager version.

[ilsaloving-gander]

I just opened a ticket for what may be the same problem, and I've tried both 1.18.3 and 1.19.1. When I create a cert resource, certmanager start spewing CR revisions as fast as it can make them.

#8234

[cert-manager-bot]

Issues go stale after 6 months of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues remain open for an additional 3 months of inactivity and then close.
If this issue is safe to close now please do so with /close.
/lifecycle stale