oauth2: cannot fetch token: Post \https://oauth2.googleapis.com/token": x509: certificate signed by unknown authority"

[yaroslavkasatikov]

Hello team,
I trying to configure cert-manager with ACME cluster-issuer and smallstepCA. All worked fine with http01 solver but decided to try checking dns01 solver with GloudDNS (private zone which solved from cluster VPC correctly).

ClusterIssuer:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: smallstep-dns
spec:
  acme:
    server: https://smallstep-step-certificates.smallstep.svc.cluster.local/acme/acme/directory
    preferredChain: "Step Online CA"
    privateKeySecretRef:
      name: acme-issuer-account-key
    solvers:
    - dns01:
        cloudDNS:
          # The ID of the GCP project
          project: sandbox-yaroslav
          # This is the secret used to access the service account
          serviceAccountSecretRef:
            name: clouddns-dns01-solver-svc-acct
            key: key.json

Certificate resource:

kind: Certificate
metadata:
  name: dns0
spec:
  secretName: dns0-secret
  issuerRef:
    name: smallstep-dns
    kind: ClusterIssuer

  dnsNames:
  - dns0.apps.privatezone

Certificates are not issue. Here is the event for Certificates resource:

  Conditions:
    Last Transition Time:        2021-01-27T16:47:57Z
    Message:                     Issuing certificate as Secret does not exist
    Reason:                      DoesNotExist
    Status:                      False
    Type:                        Ready
    Last Transition Time:        2021-01-27T16:47:57Z
    Message:                     Issuing certificate as Secret does not exist
    Reason:                      DoesNotExist
    Status:                      True
    Type:                        Issuing
  Next Private Key Secret Name:  dns0-9gwcc
Events:
  Type    Reason     Age    From          Message
  ----    ------     ----   ----          -------
  Normal  Issuing    8m18s  cert-manager  Issuing certificate as Secret does not exist
  Normal  Generated  8m18s  cert-manager  Stored new private key in temporary Secret resource "dns0-9gwcc"
  Normal  Requested  8m18s  cert-manager  Created new CertificateRequest resource "dns0-lxznb"

Here is event for CertificateRequest:

  Conditions:
    Last Transition Time:  2021-01-27T16:47:57Z
    Message:               Waiting on certificate issuance from order cert-manager/dns0-lxznb-1458637769: "pending"
    Reason:                Pending
    Status:                False
    Type:                  Ready
Events:
  Type    Reason        Age    From          Message
  ----    ------        ----   ----          -------
  Normal  OrderCreated  9m21s  cert-manager  Created Order resource cert-manager/dns0-lxznb-1458637769
  Normal  OrderPending  9m21s  cert-manager  Waiting on certificate issuance from order cert-manager/dns0-lxznb-1458637769: ""

So, I checked logs from certmanager controller pod and was surprised:

I0127 16:47:57.888280       1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "dns0-lxznb" condition "Ready" to 2021-01-27 16:47:57.888270175 +0000 UTC m=+80.158752076
I0127 16:47:57.914097       1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "dns0-lxznb" condition "Ready" to 2021-01-27 16:47:57.914069059 +0000 UTC m=+80.184550970
E0127 16:47:57.929282       1 controller.go:158] cert-manager/controller/certificaterequests-issuer-acme "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"dns0-lxznb\": the object has been modified; please apply your changes to the latest version and try again" "key"="cert-manager/dns0-lxznb"
E0127 16:47:58.789899       1 controller.go:158] cert-manager/controller/challenges "msg"="re-queuing item due to error processing" "error"="GoogleCloud API call failed: Get \"https://dns.googleapis.com/dns/v1/projects/sandbox-yaroslav/managedZones?alt=json&dnsName=aethernew.gcp.aws-openshift.club.&prettyPrint=false\": oauth2: cannot fetch token: Post \"https://oauth2.googleapis.com/token\": x509: certificate signed by unknown authority" "key"="cert-manager/dns0-lxznb-1458637769-3161905289"
E0127 16:47:58.815435       1 controller.go:158] cert-manager/controller/challenges "msg"="re-queuing item due to error processing" "error"="GoogleCloud API call failed: Get \"https://dns.googleapis.com/dns/v1/projects/sandbox-yaroslav/managedZones?alt=json&dnsName=aethernew.gcp.aws-openshift.club.&prettyPrint=false\": oauth2: cannot fetch token: Post \"https://oauth2.googleapis.com/token\": x509: certificate signed by unknown authority" "key"="cert-manager/dns0-lxznb-1458637769-3161905289"
I0127 16:48:02.433912       1 setup.go:170] cert-manager/controller/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="acme-issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="smallstep" "resource_namespace"="" "resource_version"="v1"
I0127 16:48:02.433912       1 setup.go:170] cert-manager/controller/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="acme-issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="smallstep-dns" "resource_namespace"="" "resource_version"="v1"
E0127 16:48:03.822204       1 controller.go:158] cert-manager/controller/challenges "msg"="re-queuing item due to error processing" "error"="GoogleCloud API call failed: Get \"https://dns.googleapis.com/dns/v1/projects/sandbox-yaroslav/managedZones?alt=json&dnsName=aethernew.gcp.aws-openshift.club.&prettyPrint=false\": oauth2: cannot fetch token: Post \"https://oauth2.googleapis.com/token\": x509: certificate signed by unknown authority" "key"="cert-manager/dns0-lxznb-1458637769-3161905289"
E0127 16:48:23.854474       1 controller.go:158] cert-manager/controller/challenges "msg"="re-queuing item due to error processing" "error"="GoogleCloud API call failed: Get \"https://dns.googleapis.com/dns/v1/projects/sandbox-yaroslav/managedZones?alt=json&dnsName=aethernew.gcp.aws-openshift.club.&prettyPrint=false\": oauth2: cannot fetch token: Post \"https://oauth2.googleapis.com/token\": x509: certificate signed by unknown authority" "key"="cert-manager/dns0-lxznb-1458637769-3161905289"
E0127 16:49:03.902852       1 controller.go:158] cert-manager/controller/challenges "msg"="re-queuing item due to error processing" "error"="GoogleCloud API call failed: Get \"https://dns.googleapis.com/dns/v1/projects/sandbox-yaroslav/managedZones?alt=json&dnsName=aethernew.gcp.aws-openshift.club.&prettyPrint=false\": oauth2: cannot fetch token: Post \"https://oauth2.googleapis.com/token\": x509: certificate signed by unknown authority" "key"="cert-manager/dns0-lxznb-1458637769-3161905289"
E0127 16:50:23.964857       1 controller.go:158] cert-manager/controller/challenges "msg"="re-queuing item due to error processing" "error"="GoogleCloud API call failed: Get \"https://dns.googleapis.com/dns/v1/projects/sandbox-yaroslav/managedZones?alt=json&dnsName=aethernew.gcp.aws-openshift.club.&prettyPrint=false\": oauth2: cannot fetch token: Post \"https://oauth2.googleapis.com/token\": x509: certificate signed by unknown authority" "key"="cert-manager/dns0-lxznb-1458637769-3161905289"
E0127 16:53:03.996629       1 controller.go:158] cert-manager/controller/challenges "msg"="re-queuing item due to error processing" "error"="GoogleCloud API call failed: Get \"https://dns.googleapis.com/dns/v1/projects/sandbox-yaroslav/managedZones?alt=json&dnsName=aethernew.gcp.aws-openshift.club.&prettyPrint=false\": oauth2: cannot fetch token: Post \"https://oauth2.googleapis.com/token\": x509: certificate signed by unknown authority" "key"="cert-manager/dns0-lxznb-1458637769-3161905289"

So it's strange that I got error x509: certificate signed by unknown authority

I couldn't execute commands in pod because there is no shell inside.

Could you please give me advise what goes wrong?

Teseted on cert-manager 1.0.4 and 1.1.0.

Openshift (OKD) version:
Client Version: 4.6.0-0.okd-2020-11-27-200126
Server Version: 4.6.0-0.okd-2020-11-27-200126
Kubernetes Version: v1.19.0-rc.2.1077+43983cda8af930-dirty

[jetstack-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

[jetstack-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle rotten
/remove-lifecycle stale

[jetstack-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to jetstack.
/close

[jetstack-bot]

@jetstack-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to jetstack.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.