[Bug] Issue migrating from kube-lego - Order resource not owned by this CertificateRequest, retrying

[zquestz]

Went through the tutorial here -> https://cert-manager.io/docs/tutorials/acme/migrating-from-kube-lego/ however it didn't seem to work, as I am getting the error "re-queuing item due to error processing" "error"="found Order resource not owned by this CertificateRequest, retrying". I deployed into the cert-manager namespace with:

Downloads/kube » helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --version v0.13.1

I then went through and double checked things in the tutorial.

1. Kube lego is at 0 pods.

Downloads/kube » kubectl get pods --namespace kube-lego
No resources found in kube-lego namespace.

2. Cluster issuer seems to be working fine.

Downloads/kube » kubectl describe clusterissuer letsencrypt-prod   
Name:         letsencrypt-prod
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1alpha2
Kind:         ClusterIssuer
Metadata:
  Creation Timestamp:  2020-03-10T19:13:40Z
  Generation:          1
  Resource Version:    266508999
  Self Link:           /apis/cert-manager.io/v1alpha2/clusterissuers/letsencrypt-prod
  UID:                 d88167c6-7c8d-4e44-825e-2ed065399d2b
Spec:
  Acme:
    Email:  josh@visionati.com
    Private Key Secret Ref:
      Name:  letsencrypt-private-key
    Server:  https://acme-v02.api.letsencrypt.org/directory
    Solvers:
      http01:
        Ingress:
          Class:  nginx
Status:
  Acme:
    Last Registered Email:  <MY EMAIL>
    Uri:                    https://acme-v02.api.letsencrypt.org/acme/acct/<MY ACCT>
  Conditions:
    Last Transition Time:  2020-03-10T19:13:40Z
    Message:               The ACME account was registered with the ACME server
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready
Events:                    <none>

3. I then updated my ingress shim with.

helm upgrade cert-manager \
     jetstack/cert-manager \
     --namespace cert-manager \
     --set ingressShim.defaultIssuerName=letsencrypt-prod \
     --set ingressShim.defaultIssuerKind=ClusterIssuer

4. All the certs showed up... but none are ready!

NAMESPACE   NAME                      READY   SECRET                    AGE
default     acceptbitcoincash-tls     False   acceptbitcoincash-tls     22m
default     bchd-web-tls              False   bchd-web-tls              22m
default     cashaddress-tls           False   cashaddress-tls           22m
default     cashshuffle-tls           False   cashshuffle-tls           22m
default     emlink-tls                False   emlink-tls                22m
default     ipfs-tls                  False   ipfs-tls                  22m
default     jumps-tls                 False   jumps-tls                 22m
default     neutrino-tls              False   neutrino-tls              22m
default     txhighway-tls             False   txhighway-tls             22m
default     visagoapi-tls             False   visagoapi-tls             22m
default     visionati-demo-tls        False   visionati-demo-tls        22m
default     visionati-marketing-tls   False   visionati-marketing-tls   22m

5. Logs show failures ... lots of them.

Downloads/kube » kubectl logs -n cert-manager -l app=cert-manager -c cert-manager
E0310 19:37:24.641324       1 controller.go:131] cert-manager/controller/certificaterequests-issuer-acme "msg"="re-queuing item  due to error processing" "error"="found Order resource not owned by this CertificateRequest, retrying" "key"="default/acceptbitcoincash-tls-1554461761" 
I0310 19:37:24.820287       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-acme "msg"="syncing item" "key"="default/bchd-web-tls-3441295704" 
E0310 19:37:24.820962       1 sync.go:135] cert-manager/controller/certificaterequests-issuer-acme "msg"="error issuing certificate request" "error"="found Order resource not owned by this CertificateRequest, retrying" "related_resource_kind"="ClusterIssuer" "related_resource_name"="letsencrypt-prod" "related_resource_namespace"="" "resource_kind"="CertificateRequest" "resource_name"="bchd-web-tls-3441295704" "resource_namespace"="default" 
E0310 19:37:24.821007       1 controller.go:131] cert-manager/controller/certificaterequests-issuer-acme "msg"="re-queuing item  due to error processing" "error"="found Order resource not owned by this CertificateRequest, retrying" "key"="default/bchd-web-tls-3441295704" 
I0310 19:37:26.016522       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-acme "msg"="syncing item" "key"="default/cashshuffle-tls-258628198" 
E0310 19:37:26.017072       1 sync.go:135] cert-manager/controller/certificaterequests-issuer-acme "msg"="error issuing certificate request" "error"="found Order resource not owned by this CertificateRequest, retrying" "related_resource_kind"="ClusterIssuer" "related_resource_name"="letsencrypt-prod" "related_resource_namespace"="" "resource_kind"="CertificateRequest" "resource_name"="cashshuffle-tls-258628198" "resource_namespace"="default" 
E0310 19:37:26.017118       1 controller.go:131] cert-manager/controller/certificaterequests-issuer-acme "msg"="re-queuing item  due to error processing" "error"="found Order resource not owned by this CertificateRequest, retrying" "key"="default/cashshuffle-tls-258628198" 
I0310 19:37:26.217236       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-acme "msg"="syncing item" "key"="default/visagoapi-tls-914857347" 
E0310 19:37:26.217918       1 sync.go:135] cert-manager/controller/certificaterequests-issuer-acme "msg"="error issuing certificate request" "error"="found Order resource not owned by this CertificateRequest, retrying" "related_resource_kind"="ClusterIssuer" "related_resource_name"="letsencrypt-prod" "related_resource_namespace"="" "resource_kind"="CertificateRequest" "resource_name"="visagoapi-tls-914857347" "resource_namespace"="default" 
E0310 19:37:26.217970       1 controller.go:131] cert-manager/controller/certificaterequests-issuer-acme "msg"="re-queuing item  due to error processing" "error"="found Order resource not owned by this CertificateRequest, retrying" "key"="default/visagoapi-tls-914857347" 

Expected behaviour:
I would expect not to see errors in the logs, and certs renew without any issues.

Steps to reproduce the bug:
Went through the steps in the url above.

Anything else we need to know?:
I think that is everything.

Environment details::

• Kubernetes version (e.g. v1.10.2): v1.15.9-gke.12
• Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): GKE
• cert-manager version (e.g. v0.4.0): v0.13.1
• Install method (e.g. helm or static manifests): Helm

/kind bug

[zquestz]

Closing, figured out the issue. There were some stale acme.cert-manager.io order and challenge requests from an attempted upgrade 4 months ago... clearing those out led to everything working!