Certificate not renewing, referencing unknown order?

[thanatos]

Describe the bug:

I have an Ingress that is using cert-manager to issue a certificate; the Cert:

apiVersion: cert-manager.io/v1beta1
kind: Certificate
metadata:
  creationTimestamp: "2020-08-21T23:00:03Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
  name: keycloak-cert
  namespace: keycloak
  ownerReferences:
  - apiVersion: extensions/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: Ingress
    name: keycloak
    uid: 949c857b-5193-4292-a951-42839891ee91
  resourceVersion: "40100674"
  selfLink: /apis/cert-manager.io/v1beta1/namespaces/keycloak/certificates/keycloak-cert
  uid: de4f350f-3e2c-40bf-af8b-b78d93b612d2
spec:
  dnsNames:
  - keycloak.example.com
  issuerRef:
    group: cert-manager.io
    kind: ClusterIssuer
    name: lets-encrypt
  secretName: keycloak-cert
status:
  conditions:
  - lastTransitionTime: "2020-12-01T06:23:47Z"
    message: Certificate expired on Thu, 19 Nov 2020 22:14:46 UTC
    reason: Expired
    status: "False"
    type: Ready
  - lastTransitionTime: "2020-12-02T03:23:48Z"
    message: 'The certificate request has failed to complete and will be retried:
      Failed to wait for order resource "keycloak-cert-rdvxw-332247529" to become
      ready: order is in "errored" state: Failed to retrieve Order resource: 404 urn:ietf:params:acme:error:malformed:
      No order for ID 5789985503'
    reason: Failed
    status: "False"
    type: Issuing
  lastFailureTime: "2020-12-02T03:23:48Z"
  notAfter: "2020-11-19T22:14:46Z"
  notBefore: "2020-08-21T22:14:46Z"
  renewalTime: "2020-10-20T22:14:46Z"
  revision: 1

As you can see by the notAfter or the renwalTime, very expired. (I've anonymized the domain; it's not really for example.com.)

The order it references in status doesn't exist:

» k get orders -n keycloak                           
NAME                                  STATE     AGE
keycloak-login-cert-4rpbn-984470225   valid     102d
keycloak-login-cert-c6jsq-984470225   errored   42d

(those are to a different Certificate, which is also having the same problem.) I deleted the Order, since it seemed like it was wedged. It hadn't been able to verify the domain, and by the time I got to fixing it, the ACME order was expired, I think. So I deleted the Order since usually that causes cert-manager to just re-create it, and effectively, retry.

There are no Challenges for this cert, either:

NAME                                           STATE     DOMAIN                    AGE
keycloak-login-cert-c6jsq-984470225-99017855   expired   login.example.com   42d

At this point, I've also tried restarting cert-manager, and still, nothing. The lastTransitionTime on the status keeps updating too, but I have no idea where it's getting that Order?

Honestly, I'm not sure if it's cert-manager or me. I've failed at ACME a few times, and usually deleting orders/challenges is sufficient to get cert-manager to retry, once I've corrected the issue that is causing ACME to not happen.

Expected behaviour: The cert to renew, eventually, of course.
Steps to reproduce the bug: I got into this state by having the domain name pointed at the wrong IP for a while, overlapping a renewal. (ISP outage resulted in an IP change.)

Anything else we need to know?: It might very well be PEBKAC.

Environment details::

• Kubernetes version: 1.16.8
• Cloud-provider/provisioner: none
• cert-manager version: 0.16.1
• Install method: helm

/kind bug

[meyskens]

404 urn:ietf:params:acme:error:malformed: comes from Let's Encrypt

What do you get if you describe the orders?

[thanatos]

So, for keycloak, I deleted the Orders, hoping that that would kick cert-manager out of its funk. For one of the ones that still exists,
(test-ingress)

  Reason:           Failed to retrieve Order resource: 404 urn:ietf:params:acme:error:malformed: No order for ID 5675067672

Which sort of makes sense; I think LE has probably deleted the order on their side, since I wasn't passing the challenges. It took me some time to notice & correct the issue w/ DNS, and I think in that time, the order likely expired. That's my fault, and why I deleted the order for the keycloak cert, since I figured that that was what had happened (it had expired) and that by deleting it, cert-manager would recreate it & then subsequently get a new ACME order, a new challenge, and re-issue the challenge but this time w/ working DNS records.

[PSanetra]

@meyskens is there some way to trigger cert-manager to create a new order, when an already existing order has errored?

[thanatos]

So, I ended up kubectl delete cert <name of cert> here for the failing certificates. Since they were "ingress" certs, (from annotations on an Ingress) it seems like cert-manager noticed the delete and just immediately re-created them, and from there, attempted to get a cert & was successful.

So, not a great resolution. We'll see what happens at my next renewal in ~60d, but I expect it'll be fine. (So long as my DNS remains pointed at the right IP this time…)

[rcanderson23]

Ran into a similar issue recently. To resolve this without deleting the certificate, delete the certificaterequest(s) that are False. This will force cert-manager to create a new certificaterequest during the next sync period and should resolve the issue.

[wallrj]

@irbekrm Might this be fixed by #4130 ? If so, stay tuned for cert-manager 1.5 in August.

[irbekrm]

Ran into a similar issue recently. To resolve this with out deleting the certificate, delete the certificaterequest(s) that are False. This will force cert-manager to create a new certificaterequest during the next sync period and should resolve the issue.

This does sound like #4130 👍🏼

In case of the original issue, it may have been interesting to look at the state of the latest CertificateRequest if that's not re-tried then that too might be #4130

[arianitu]

I just ran into this as well, I deleted the Certificate and it successfully generated. I'm going to try to upgrade cert-manager

[PhilipSchmid]

Any update on this guys? Did Cert-Manager 1.5.X solve this issue for you? Thanks.

[kquinsland]

Any update on this guys? Did Cert-Manager 1.5.X solve this issue for you? Thanks.

I was migrating from 1.3.0 to 1.6 using the static install method. (stopping along the way at 1.4, 1.5).
I waited about 120s from the kubectl apply -f ... download/v1.{4,5,6}.0/cert-manager.yaml commands. I don't know if i should have waited longer or not between upgrades but about 20min after upgrading to 1.6, I checked on the status of a few certificates and noticed that they still had not been renewed. That's when I noticed the error mentioned in this thread. A quick google brought me here.

I had to manually delete and force a renew:

❯ kubectl -n kube-system delete certificaterequest/k8s-dash<...>
< ... repeat for a few more certs in a few more name spaces ... >

❯ cmctl renew --all -A
Manually triggered issuance of Certificate kube-system/k8s-dash<...>
< ... a few more lines ...>

Here's a snip of the logs from the controller pod after deploying 1.6:

❯ kubectl -n cert-manager logs cert-manager-85df969c8c-mwmkk

I1105 17:07:40.878424       1 util.go:84] cert-manager/controller/orders/handleOwnedResource 
"msg"="owning resource not found in cache" "related_resource_kind"="Order" "related_resource_name"="k8s-dash-cert-<...>" "related_resource_namespace"="kube-system" "resource_kind"="Challenge" "resource_name"="k8s-dash-cert-<...>-<...>" "resource_namespace"="kube-system" "resource_version"="v1" 

I1105 17:07:41.352232       1 util.go:84] cert-manager/controller/certificaterequests/handleOwnedResource "msg"="owning resource not found in cache" 
"related_resource_kind"="CertificateRequest" "related_resource_name"="k8s-dash-cert-<...>" "related_resource_namespace"="kube-system" "resource_kind"="Order" "resource_name"="k8s-dash-cert-<...>-<...>" "resource_namespace"="kube-system" "resource_version"="v1" 

I1105 17:07:41.353084       1 util.go:84] cert-manager/controller/certificaterequests/handleOwnedResource "msg"="owning resource not found in cache" 
"related_resource_kind"="CertificateRequest" "related_resource_name"="k8s-dash-cert-9fjv6" "related_resource_namespace"="kube-system" "resource_kind"="Order" "resource_name"="k8s-dash-cert-<...>-<...>" "resource_namespace"="kube-system" "resource_version"="v1" 

I1105 17:07:41.353535       1 util.go:84] cert-manager/controller/certificaterequests/handleOwnedResource "msg"="owning resource not found in cache" 
"related_resource_kind"="CertificateRequest" "related_resource_name"="k8s-dash-cert-<...>" "related_resource_namespace"="kube-system" "resource_kind"="Order" "resource_name"="k8s-dash-cert-<...>-<...>" "resource_namespace"="kube-system" "resource_version"="v1" 

I1105 17:07:41.353760       1 util.go:84] cert-manager/controller/certificaterequests/handleOwnedResource "msg"="owning resource not found in cache" 
"related_resource_kind"="CertificateRequest" "related_resource_name"="k8s-dash-cert-cjdrk" "related_resource_namespace"="kube-system" "resource_kind"="Order" "resource_name"="k8s-dash-cert-<...>" "resource_namespace"="kube-system" "resource_version"="v1" 

I1105 17:07:41.616909       1 trigger_controller.go:160] cert-manager/controller/certificates-trigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="kube-system/k8s-dash-cert" "retry_delay"=3225383151743

It may be that the 1.5 release tried to address the issue but didn't have time to finish before I replaced with 1.6... That is the only explanation I can come up with for the last log line about waiting to retry....

[irbekrm]

@kquinsland this seems like a different issue

"msg"="owning resource not found in cache" "related_resource_kind"="Order" "related_resource_name"="k8s-dash-cert-<...>" "related_resource_namespace"="kube-system" "resource_kind"="Challenge" "resource_name"="k8s-dash-cert-<...>-<...>" "resource_namespace"="kube-system"

This message can come up right after the controller pod has started, before all caches have fully synced as part of the normal flow.

I1105 17:07:41.616909 1 trigger_controller.go:160] cert-manager/controller/certificates-trigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="kube-system/k8s-dash-cert" "retry_delay"=3225383151743

This suggests that there is already a failed CertificateRequest for the kube-system/k8s-dash-cert Certificate, so cert-manager is backing off for an hour and will retry to renew after that. It would probably make sense to look into why the CertificateRequest failed if that is the case