Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cloudflare DNS resolver fails: Error: 6003: Invalid request headers #3021

Closed
ltetrel opened this issue Jun 18, 2020 · 38 comments
Closed

Cloudflare DNS resolver fails: Error: 6003: Invalid request headers #3021

ltetrel opened this issue Jun 18, 2020 · 38 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. triage/needs-information Indicates an issue needs more information in order to work on it.

Comments

@ltetrel
Copy link

ltetrel commented Jun 18, 2020

Describe the bug:
Cannot get DNS resolver to work with cloudflare account

cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Cloudflare API Error \n\t Error: 6003: Invalid request headers\u003c- 6111: Invalid format for Authorization header"

Expected behaviour:
The challenge to be accepted

Steps to reproduce the bug:
I strictly followed the documentation with an api-token:
https://cert-manager.io/docs/configuration/acme/dns01/cloudflare/#api-tokens
Tried also with the global api-key.

Anything else we need to know?:
The HTTP does not work neither.

"msg"="propagation check failed" "error"="wrong status code '404', expected '200'" "type"="http-01"

Environment details::

  • Kubernetes version (e.g. v1.10.2): v1.18.3
  • Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): bare-metal opennebula
  • cert-manager version (e.g. v0.4.0): 0.15.1
  • Install method (e.g. helm or static manifests): helm

/kind bug

@jetstack-bot jetstack-bot added the kind/bug Categorizes issue or PR as related to a bug. label Jun 18, 2020
@keslerm
Copy link

keslerm commented Jun 19, 2020

Getting this too with API keys as well as the API tokens being broken

@rklubenspies
Copy link

Also experiencing this, noticing this error:

Status:
  Presented:   false
  Processing:  true
  Reason:      Cloudflare API Error
                Error: 6003: Invalid request headers<- 6103: Invalid format for X-Auth-Key header
  State:       pending

@jmgilman
Copy link

Did anyone resolve this issue yet? I'm having to manually renew expiring certs and its not fun.

@luishdez
Copy link

luishdez commented Jun 25, 2020

Yep this is broken. With global and origin key.

I0625 06:21:26.945303       1 logger.go:149] Calling DNS01ChallengeRecord
I0625 06:21:26.945424       1 sync.go:179] cert-manager/controller/orders "msg"="No action taken" "resource_kind"="Order" "resource_name"="cert-isent-co-113337196-2722511178" "resource_namespace"="default" 
I0625 06:21:26.945478       1 controller.go:147] cert-manager/controller/orders "msg"="finished processing work item" "key"="default/cert-isent-co-113337196-2722511178" 
I0625 06:21:28.432671       1 controller.go:141] cert-manager/controller/challenges "msg"="syncing item" "key"="default/cert-instasent-com-2799934583-3189091394-3676235352" 
I0625 06:21:28.433044       1 dns.go:106] cert-manager/controller/challenges/Present "msg"="presenting DNS01 challenge for domain" "dnsName"="instasent.com" "domain"="instasent.com" "resource_kind"="Challenge" "resource_name"="cert-instasent-com-2799934583-3189091394-3676235352" "resource_namespace"="default" "type"="dns-01" 
I0625 06:21:28.587134       1 controller.go:141] cert-manager/controller/challenges "msg"="syncing item" "key"="default/cert-isent-co-113337196-2722511178-1495611924" 
I0625 06:21:28.587426       1 dns.go:106] cert-manager/controller/challenges/Present "msg"="presenting DNS01 challenge for domain" "dnsName"="isent.co" "domain"="isent.co" "resource_kind"="Challenge" "resource_name"="cert-isent-co-113337196-2722511178-1495611924" "resource_namespace"="default" "type"="dns-01" 
E0625 06:21:28.637534       1 controller.go:143] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Cloudflare API Error \n\t Error: 6003: Invalid request headers\u003c- 6111: Invalid format for Authorization header" "key"="default/cert-instasent-com-2799934583-3189091394-3676235352" 
E0625 06:21:28.799572       1 controller.go:143] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Cloudflare API Error \n\t Error: 6003: Invalid request headers\u003c- 6111: Invalid format for Authorization header" "key"="default/cert-isent-co-113337196-2722511178-1495611924" 
I0625 06:21:48.637843       1 controller.go:141] cert-manager/controller/challenges "msg"="syncing item" "key"="default/cert-instasent-com-2799934583-3189091394-3676235352" 
I0625 06:21:48.638227       1 dns.go:106] cert-manager/controller/challenges/Present "msg"="presenting DNS01 challenge for domain" "dnsName"="instasent.com" "domain"="instasent.com" "resource_kind"="Challenge" "resource_name"="cert-instasent-com-2799934583-3189091394-3676235352" "resource_namespace"="default" "type"="dns-01" 
I0625 06:21:48.799777       1 controller.go:141] cert-manager/controller/challenges "msg"="syncing item" "key"="default/cert-isent-co-113337196-2722511178-1495611924" 
I0625 06:21:48.800242       1 dns.go:106] cert-manager/controller/challenges/Present "msg"="presenting DNS01 challenge for domain" "dnsName"="isent.co" "domain"="isent.co" "resource_kind"="Challenge" "resource_name"="cert-isent-co-113337196-2722511178-1495611924" "resource_namespace"="default" "type"="dns-01" 
E0625 06:21:48.834186       1 controller.go:143] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Cloudflare API Error \n\t Error: 6003: Invalid request headers\u003c- 6111: Invalid format for Authorization header" "key"="default/cert-instasent-com-2799934583-3189091394-3676235352" 
E0625 06:21:48.995882       1 controller.go:143] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Cloudflare API Error \n\t Error: 6003: Invalid request headers\u003c- 6111: Invalid format for Authorization header" "key"="default/cert-isent-co-113337196-2722511178-1495611924" 

@rklubenspies
Copy link

For what it's worth, I found and followed this article and somehow this started working again for me. Not sure what was done differently in this article than what I was doing, but it started working for me 🤷‍♂️

@ltetrel
Copy link
Author

ltetrel commented Jul 1, 2020

@rklubenspies
So not sure but maybe it is related to the fact that you set:
kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true

which is not on the official documentation..

I had previous issues with the webhook validation but resolved it (#2918 (comment)), maybe it is relate to that 🤨

@rklubenspies
Copy link

@ltetrel I think this might have something to do with it—I didn't notice this until you pointed it out, but this is definitely something I didn't set the first time around.

@ltetrel
Copy link
Author

ltetrel commented Jul 2, 2020

Still having the same issue even after strictly following https://blog.darkedges.com/2020/05/04/cert-manager-kubernetes-cloudflare-dns-update/ ..

@rklubenspies
Copy link

@ltetrel Another thing I had to do was restart all the cert-manager resources after making some configuration changes before it would properly process the DNS challenges:

% kubectl get pods -n cert-manager

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-9b8969d86-242p4               1/1     Running   0          14h
cert-manager-cainjector-8545fdf87c-zqkmd   1/1     Running   0          14h
cert-manager-webhook-8c5db9fb6-b77s9       1/1     Running   0          14h

% kubectl delete pod -n cert-manager cert-manager-9b8969d86-242p4
% kubectl delete pod -n cert-manager cert-manager-cainjector-8545fdf87c-zqkmd
% kubectl delete pod -n cert-manager cert-manager-webhook-8c5db9fb6-b77s9

@meyskens
Copy link
Contributor

meyskens commented Aug 4, 2020

Is everything solved now?

/triage needs-information

@jetstack-bot jetstack-bot added the triage/needs-information Indicates an issue needs more information in order to work on it. label Aug 4, 2020
@ltetrel
Copy link
Author

ltetrel commented Sep 3, 2020

Hi,

Sadly no. I rely on a certificate I manually asked from certbot and injected into the k8s cluster for now.
I am also trying with an http challenge without more success. This can maybe causes by our network configuration, I will let you know..
Does the DNS challenge need access to the domain (something on our server), or it is just relying on the provider server (cloudflare in our case) ?

@meyskens
Copy link
Contributor

meyskens commented Sep 8, 2020

It needs valid DNS resolvers to look up the SOA record to find the zone to use in Cloudflare

@gitnik
Copy link

gitnik commented Oct 1, 2020

Running into the exact same issue. Disabling the validation did not fix this

@gitnik
Copy link

gitnik commented Oct 1, 2020

OK so using a new API Token instead of the global one works, as we had set apiTokenSecretRef

@luishdez
Copy link

luishdez commented Oct 2, 2020

I did fixed with global token using

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-key
  namespace: cert-manager
type: Opaque
stringData:
  api-token: xxxxxxxxxx
----
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: test@test.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - dns01:
        cloudflare:
          email: test@test.com
          apiKeySecretRef:
            name: cloudflare-api-key
            key: api-token

@ltetrel
Copy link
Author

ltetrel commented Oct 7, 2020

I am running v1.0.2 and was able to get a valid certificate with a HTTP01 challenge.
Did not try the DNS01 challenge but it could be worth that someone check it with the newest cert-manager version.

@timothyclarke
Copy link

I'm using cert-manager v0.12.0. This is working when I use the Global API key from the admin account.
I'd prefer to swap over to apiTokenSecretRef and use a more restricted API Token, however I keep getting * spec.acme.solvers.dns01.cloudflare.apiKeySecretRef: Required value
I'm using apiVersion: cert-manager.io/v1alpha2

@meyskens
Copy link
Contributor

@timothyclarke this got solved in newer releases of cert-manager

@oliverbaehler
Copy link

oliverbaehler commented Nov 17, 2020

@ltetrel I am currently running cert-manager v1.0.4 (Kubernetes v1.19.4) and still get the same behavior.

Creating ClusterIssuer:

---
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret 
  namespace: cert-manager
type: Opaque
stringData:
  api: globalAPIToken
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: sample
spec:
  acme:
    email: mymail@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: cloudflare-account-key
    solvers:
    - dns01:
        cloudflare:
          email: mymail@example.com
          apiTokenSecretRef:
            name: cloudflare-api-token-secret
            key: api

After requesting a new certificate, the certificaterequest is stuck in pending state:

Status:
  Conditions:
    Last Transition Time:  2020-11-17T16:36:57Z
    Message:               Waiting on certificate issuance from order cattle-system/sample-sz4bc-3622413353: "pending"
    Reason:                Pending
    Status:                False
    Type:                  Ready
Events:
  Type    Reason        Age   From          Message
  ----    ------        ----  ----          -------
  Normal  OrderCreated  11m   cert-manager  Created Order resource cattle-system/sample-sz4bc-3622413353

And the challange has the error:

Status:
  Presented:   false
  Processing:  true
  Reason:      Cloudflare API Error for GET "/zones?name=myzone.ch" 
                Error: 6003: Invalid request headers<- 6111: Invalid format for Authorization header
  State:       pending
Events:
  Type     Reason        Age                  From          Message
  ----     ------        ----                 ----          -------
  Normal   Started       13m                  cert-manager  Challenge scheduled for processing
  Warning  PresentError  2m38s (x8 over 13m)  cert-manager  Error presenting challenge: Cloudflare API Error for GET "/zones?name=myzone.ch" 
            Error: 6003: Invalid request headers<- 6111: Invalid format for Authorization header

Controller Logs during this Workflow (Certificate Creation):

E1117 17:36:14.468887       1 requestmanager_controller.go:127] cert-manager/controller/CertificateRequestManager "msg"="certificate not found for key" "error"="certificate.cert-manager.io \"buttahtoast\" not found" "key"="cattle-system/buttahtoast" 
E1117 17:36:14.469173       1 trigger_controller.go:142] cert-manager/controller/CertificateTrigger "msg"="certificate not found for key" "error"="certificate.cert-manager.io \"buttahtoast\" not found" "key"="cattle-system/buttahtoast" 
E1117 17:36:14.469260       1 keymanager_controller.go:137] cert-manager/controller/CertificateKeyManager "msg"="certificate not found for key" "error"="certificate.cert-manager.io \"buttahtoast\" not found" "key"="cattle-system/buttahtoast" 
E1117 17:36:14.469888       1 readiness_controller.go:130] cert-manager/controller/CertificateReadiness "msg"="certificate not found for key" "error"="certificate.cert-manager.io \"buttahtoast\" not found" "key"="cattle-system/buttahtoast" 
E1117 17:36:14.470115       1 issuing_controller.go:152] cert-manager/controller/CertificateIssuing "msg"="certificate not found for key" "error"="certificate.cert-manager.io \"buttahtoast\" not found" "key"="cattle-system/buttahtoast" 
I1117 17:36:14.831039       1 conditions.go:173] Setting lastTransitionTime for Certificate "buttahtoast" condition "Issuing" to 2020-11-17 17:36:14.831023665 +0000 UTC m=+4255.315979130
I1117 17:36:14.832117       1 conditions.go:173] Setting lastTransitionTime for Certificate "buttahtoast" condition "Ready" to 2020-11-17 17:36:14.832109032 +0000 UTC m=+4255.317064539
E1117 17:36:15.051840       1 controller.go:158] cert-manager/controller/CertificateTrigger "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"buttahtoast\": the object has been modified; please apply your changes to the latest version and try again" "key"="cattle-system/buttahtoast" 
I1117 17:36:15.051942       1 conditions.go:173] Setting lastTransitionTime for Certificate "buttahtoast" condition "Issuing" to 2020-11-17 17:36:15.051933985 +0000 UTC m=+4255.536889506
I1117 17:36:15.861518       1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "buttahtoast-pk846" condition "Ready" to 2020-11-17 17:36:15.861506304 +0000 UTC m=+4256.346461717
E1117 17:36:15.913175       1 controller.go:184] cert-manager/controller/certificaterequests-issuer-ca "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"buttahtoast-gnbtb\" not found"  
E1117 17:36:15.913235       1 controller.go:184] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"buttahtoast-gnbtb\" not found"  
E1117 17:36:15.913335       1 controller.go:184] cert-manager/controller/certificaterequests-issuer-vault "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"buttahtoast-gnbtb\" not found"  
E1117 17:36:15.913619       1 controller.go:184] cert-manager/controller/certificaterequests-issuer-acme "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"buttahtoast-gnbtb\" not found"  
E1117 17:36:15.913864       1 controller.go:184] cert-manager/controller/certificaterequests-issuer-venafi "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"buttahtoast-gnbtb\" not found"  
E1117 17:36:16.155564       1 controller.go:142] cert-manager/controller/orders "msg"="order in work queue no longer exists" "error"="order.acme.cert-manager.io \"buttahtoast-gnbtb-3622413353\" not found"  
E1117 17:36:16.155588       1 util.go:71] cert-manager/controller/certificaterequests/handleOwnedResource "msg"="error getting referenced owning resource" "error"="certificaterequest.cert-manager.io \"buttahtoast-gnbtb\" not found" "related_resource_kind"="CertificateRequest" "related_resource_name"="buttahtoast-gnbtb" "related_resource_namespace"="cattle-system" "resource_kind"="Order" "resource_name"="buttahtoast-gnbtb-3622413353" "resource_namespace"="cattle-system" "resource_version"="v1" 
E1117 17:36:16.313696       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"buttahtoast-gnbtb-3622413353\" not found" "related_resource_kind"="Order" "related_resource_name"="buttahtoast-gnbtb-3622413353" "related_resource_namespace"="cattle-system" "resource_kind"="Challenge" "resource_name"="buttahtoast-gnbtb-3622413353-2047870714" "resource_namespace"="cattle-system" "resource_version"="v1" 
E1117 17:36:17.373377       1 sync.go:287] cert-manager/controller/challenges/finalizer "msg"="error cleaning up challenge" "error"="Cloudflare API Error for GET \"/zones?name=myzone.ch\" \n\t Error: 6003: Invalid request headers<- 6111: Invalid format for Authorization header" "dnsName"="rancher.myzone.ch" "resource_kind"="Challenge" "resource_name"="buttahtoast-gnbtb-3622413353-2047870714" "resource_namespace"="cattle-system" "resource_version"="v1" "type"="DNS-01" 
E1117 17:36:17.802316       1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"buttahtoast-gnbtb-3622413353\" not found" "related_resource_kind"="Order" "related_resource_name"="buttahtoast-gnbtb-3622413353" "related_resource_namespace"="cattle-system" "resource_kind"="Challenge" "resource_name"="buttahtoast-gnbtb-3622413353-2047870714" "resource_namespace"="cattle-system" "resource_version"="v1" 
E1117 17:36:17.824646       1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"buttahtoast-gnbtb-3622413353-2047870714\" not found"  
E1117 17:36:20.149917       1 controller.go:158] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Cloudflare API Error for GET \"/zones?name=myzone.ch\" \n\t Error: 6003: Invalid request headers<- 6111: Invalid format for Authorization header" "key"="cattle-system/buttahtoast-pk846-3622413353-2047870714" 
E1117 17:36:20.993013       1 controller.go:158] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Cloudflare API Error for GET \"/zones?name=myzone.ch\" \n\t Error: 6003: Invalid request headers<- 6111: Invalid format for Authorization header" "key"="cattle-system/buttahtoast-pk846-3622413353-2047870714" 
E1117 17:36:26.007599       1 controller.go:158] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Cloudflare API Error for GET \"/zones?name=myzone.ch\" \n\t Error: 6003: Invalid request headers<- 6111: Invalid format for Authorization header" "key"="cattle-system/buttahtoast-pk846-3622413353-2047870714" 
E1117 17:36:46.820924       1 controller.go:158] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="Cloudflare API Error for GET \"/zones?name=myzone.ch\" \n\t Error: 6003: Invalid request headers<- 6111: Invalid format for Authorization header" "key"="cattle-system/buttahtoast-pk846-3622413353-2047870714" 

Not quiet sure what's causing this, I will keep on trying and let you know if I find any solution to it.

@oliverbaehler
Copy link

siggghhh, I tried different reconfigurations etc. but as it turns out the 5th new generated Cloudflare API Token somehow worked. I don't think it really made a difference that I started double quoting the String in the secret. Maybe someone else encountering this issue can confirm, that a new, correct API Token resolves this issue. Cert-Manager documented which access roles are required: https://cert-manager.io/docs/configuration/acme/dns01/cloudflare/#api-tokens. For my part, idk what the problem with my first attempts was...

@czystyl
Copy link

czystyl commented Dec 4, 2020

I'm facing a similar problem.

Status:
  Presented:   false
  Processing:  true
  Reason:      Cloudflare API Error for GET "/zones?name=xxx.dev"
                Error: 6003: Invalid request headers<- 6103: Invalid format for X-Auth-Key header
  State:       pending
Events:
  Type     Reason        Age                 From          Message
  ----     ------        ----                ----          -------
  Normal   Started       103s                cert-manager  Challenge scheduled for processing
  Warning  PresentError  37s (x5 over 102s)  cert-manager  Error presenting challenge: Cloudflare API Error for GET "/zones?name=xxx.dev"
            Error: 6003: Invalid request headers<- 6103: Invalid format for X-Auth-Key header

I tried to use api-token and api-key but without success.

@xlanor
Copy link

xlanor commented Dec 31, 2020

@luishdez which version of certmanager are you running?

I'm on the latest version of certmanager, your yaml simply results in Error: 6003: Invalid request headers<- 6103: Invalid format for X-Auth-Key header

@xlanor
Copy link

xlanor commented Dec 31, 2020

I was just able to deploy it using an api token generated from cloudflare.

Follow the steps here :
https://blog.darkedges.com/2020/05/04/cert-manager-kubernetes-cloudflare-dns-update/

but, instead of using apiKeySecretRef, using a token here, use apiTokenSecretRef.

You do not need to base64 encode the secret. I read somewhere else that they only got it to work by doing that, so I was on the wrong track.

@hadleyrich
Copy link

I've just been setting this up myself and was curious why my apiTokenSecretRef was not working. Turns out it does work if I don't base64 encode the secret. This seems odd and I don't know enough to dig through and find out why this is the case.

@hadleyrich
Copy link

I've just been setting this up myself and was curious why my apiTokenSecretRef was not working. Turns out it does work if I don't base64 encode the secret. This seems odd and I don't know enough to dig through and find out why this is the case.

Ignore me, this was my lack of understanding of the difference between data and stringData!

@Richard87
Copy link

I got the same error, but created a new token with these permissions, that worked perfectly:

All accounts - Access: Mutual TLS Certificates:Edit
All zones - Zone Settings:Edit, Zone:Edit, SSL and Certificates:Edit, DNS:Edit

I'm sure you can make a more restrictive key, but atleast these works :)

@darth-veitcher
Copy link

darth-veitcher commented Jun 15, 2021

For what it's worth I was able to fix my issue by changing from the documented Secret. Changes made:

  • Added to the cert-manager namespace
  • Used data as opposed to stringData because I base64 encoded the secret

I'm using API Token from Cloudflare with a ClusterIssuer. Didn't need to adjust from the recommended minimal permission requirements in the docs.

export API_TOKEN=$(echo -n '-myVeryS3cretAPITokenfromCloudfl4re' | base64)
cat <<EOF | kubectl apply -f - 
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: cert-manager
type: Opaque
data:
  api-token: ${API_TOKEN}
EOF

Before those changes I was seeing the below errors:

  • Secret not found (fixed by changing namespace)
  • Error: 6003: Invalid request headers<- 6111: Invalid format for Authorization header (fixed with data vs stringData)

@jetstack-bot
Copy link
Collaborator

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

@jetstack-bot jetstack-bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 16, 2021
@jetstack-bot
Copy link
Collaborator

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle rotten
/remove-lifecycle stale

@jetstack-bot jetstack-bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Oct 16, 2021
@jetstack-bot
Copy link
Collaborator

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to jetstack.
/close

@jetstack-bot
Copy link
Collaborator

@jetstack-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to jetstack.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@gaigelama
Copy link

siggghhh, I tried different reconfigurations etc. but as it turns out the 5th new generated Cloudflare API Token somehow worked. I don't think it really made a difference that I started double quoting the String in the secret. Maybe someone else encountering this issue can confirm, that a new, correct API Token resolves this issue. Cert-Manager documented which access roles are required: https://cert-manager.io/docs/configuration/acme/dns01/cloudflare/#api-tokens. For my part, idk what the problem with my first attempts was...

This got me thinking as I just had this problem on a new cluster and api key. This is happening when the cloudflare api key has a - (hyphen) in the key. My first and second key I had tried both had hyphens in them. I redeployed certmanager with the old keys to make sure it was still broken before I tried the fix and it was. I rotated the key until I had one without a hyphen and redeployed. It then worked as expected. I'd consider this a bug that needs to be fixed.

@adamzero1
Copy link

@rklubenspies So not sure but maybe it is related to the fact that you set: kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true

which is not on the official documentation..

I had previous issues with the webhook validation but resolved it (#2918 (comment)), maybe it is relate to that 🤨

Yeah looks like it was that for me, added this to my values.yaml

cert-manager:
  deploymentAnnotations:
    certmanager.k8s.io/disable-validation: "true"

Then applied and it started working

@gaigelama
Copy link

This issue seems to be closed but I don't see an associated PR with it to fix this bug (#3021 (comment))

I don't think disabling validation should be considered a fix.

@siennathesane
Copy link

I was able to resolve the issue by both not having a hyphen and by not base64 encoding the API token.

@ameyp
Copy link

ameyp commented Jun 24, 2022

Error: 6003: Invalid request headers<- 6111: Invalid format for Authorization header

Getting this in cert-manager logs, my API token doesn't have a hyphen and I'm not base64 encoding it. I'm using an API token with the minimal set of permissions listed in the documentation, and my issuer has an apiTokenSecretRef.

Edit - I had an email specified in addition to the apiTokenSecretRef, and that probably caused problems in the backend. Suggestion: ignore the email, or print a warning/error or something about invalid configuration being specified.

/reopen

@jetstack-bot
Copy link
Collaborator

@ameyp: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

Error: 6003: Invalid request headers<- 6111: Invalid format for Authorization header

Getting this in cert-manager logs, my API token doesn't have a hyphen and I'm not base64 encoding it. I'm using an API token with the minimal set of permissions listed in the documentation, and my issuer has an apiTokenSecretRef.

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@colvint
Copy link

colvint commented Jul 26, 2022

apiTokenSecretRef

For what it's worth I was able to fix my issue by changing from the documented Secret. Changes made:

  • Added to the cert-manager namespace
  • Used data as opposed to stringData because I base64 encoded the secret

I'm using API Token from Cloudflare with a ClusterIssuer. Didn't need to adjust from the recommended minimal permission requirements in the docs.

export API_TOKEN=$(echo -n '-myVeryS3cretAPITokenfromCloudfl4re' | base64)
cat <<EOF | kubectl apply -f - 
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: cert-manager
type: Opaque
data:
  api-token: ${API_TOKEN}
EOF

Before those changes I was seeing the below errors:

  • Secret not found (fixed by changing namespace)
  • Error: 6003: Invalid request headers<- 6111: Invalid format for Authorization header (fixed with data vs stringData)

Confirm changing stringData -> data in the Secret (which needs to be in cert-manager namespace) worked. Sigh the hours we burn...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. triage/needs-information Indicates an issue needs more information in order to work on it.
Projects
None yet
Development

No branches or pull requests