New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue with switching from LE staging to LE prod: 403 urn:acme:error:unauthorized: No registration exists matching provided key" #160

Open
ahmetb opened this Issue Apr 21, 2017 · 10 comments

Comments

Projects
None yet
4 participants
@ahmetb
Copy link

ahmetb commented Apr 21, 2017

I was able to successfully follow the examples/gce/README.md on Google Container Engine and have my container serving with Fake LE Root X1 CA. I saw that the Kubernetes secret was created fine too.

Then I decided to switch from LE staging to LE prod. Here's what I did and at the end you'll see the continuous error from kube-lego:

  1. kubectl delete secret echoserver-tls
  2. kubectl edit configmap/kube-lego -n kube-lego
  3. modify the url to https://acme-v01.api.letsencrypt.org/directory, save, exit
  4. kubectl delete pods --all -n kube-lego so that new config map values get picked up
  5. observe new kube-lego container started.
  6. tail logs on kube-lego container, see repeated error:
time="2017-04-21T18:49:52Z" level=warning msg="authorization failed after 1m0s: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=ng.alp.im
time="2017-04-21T18:49:52Z" level=error msg="Error while process certificate requests: no domain could be authorized successfully" context=kubelego
time="2017-04-21T18:49:52Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-04-21T18:49:52Z" level=info msg="process certificates requests for ingresses" context=kubelego
time="2017-04-21T18:49:52Z" level=info msg="creating new secret" context=secret name=echoserver-tls namespace=default
time="2017-04-21T18:49:52Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=echoserver namespace=default
time="2017-04-21T18:49:52Z" level=info msg="requesting certificate for ng.alp.im" context="ingress_tls" name=echoserver namespace=default

time="2017-04-21T18:51:05Z" level=warning msg="authorization failed after 1m0s: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=ng.alp.im
time="2017-04-21T18:51:05Z" level=error msg="Error while process certificate requests: no domain could be authorized successfully" context=kubelego
time="2017-04-21T18:51:05Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-04-21T18:51:05Z" level=info msg="process certificates requests for ingresses" context=kubelego
time="2017-04-21T18:51:05Z" level=info msg="creating new secret" context=secret name=echoserver-tls namespace=default
time="2017-04-21T18:51:05Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=echoserver namespace=default
time="2017-04-21T18:51:05Z" level=info msg="requesting certificate for ng.alp.im" context="ingress_tls" name=echoserver namespace=default

time="2017-04-21T18:52:15Z" level=warning msg="authorization failed after 1m0s: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=ng.alp.im
time="2017-04-21T18:52:15Z" level=error msg="Error while process certificate requests: no domain could be authorized successfully" context=kubelego
time="2017-04-21T18:52:15Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-04-21T18:52:15Z" level=info msg="process certificates requests for ingresses" context=kubelego
time="2017-04-21T18:52:15Z" level=info msg="creating new secret" context=secret name=echoserver-tls namespace=default
time="2017-04-21T18:52:15Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=echoserver namespace=default
time="2017-04-21T18:52:15Z" level=info msg="requesting certificate for ng.alp.im" context="ingress_tls" name=echoserver namespace=default

time="2017-04-21T18:53:26Z" level=warning msg="authorization failed after 1m0s: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=ng.alp.im
time="2017-04-21T18:53:26Z" level=error msg="Error while process certificate requests: no domain could be authorized successfully" context=kubelego
time="2017-04-21T18:53:26Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-04-21T18:53:26Z" level=info msg="process certificates requests for ingresses" context=kubelego
time="2017-04-21T18:53:26Z" level=info msg="creating new secret" context=secret name=echoserver-tls namespace=default
time="2017-04-21T18:53:26Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=echoserver namespace=default
time="2017-04-21T18:53:26Z" level=info msg="requesting certificate for ng.alp.im" context="ingress_tls" name=echoserver namespace=default
time="2017-04-21T18:54:50Z" level=warning msg="authorization failed after 1m0s: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=ng.alp.im

time="2017-04-21T18:54:50Z" level=error msg="Error while process certificate requests: no domain could be authorized successfully" context=kubelego
time="2017-04-21T18:54:50Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-04-21T18:54:50Z" level=info msg="process certificates requests for ingresses" context=kubelego
time="2017-04-21T18:54:50Z" level=info msg="creating new secret" context=secret name=echoserver-tls namespace=default
time="2017-04-21T18:54:50Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=echoserver namespace=default
time="2017-04-21T18:54:50Z" level=info msg="requesting certificate for ng.alp.im" context="ingress_tls" name=echoserver namespace=default

...

Am I doing something wrong?

@ahmetb

This comment has been minimized.

Copy link
Author

ahmetb commented Apr 21, 2017

I actually started from scratch, and when I use https://acme-v01.api.letsencrypt.org/directory to configure kube-lego initally, I always get this error:

time="2017-04-21T20:18:32Z" level=info msg="requesting certificate for n.alp.im" context="ingress_tls" name=myngx namespace=default
time="2017-04-21T20:19:47Z" level=warning msg="authorization failed after 1m0s: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im

The moment I switch to staging back and restart the kube-lego pod (and change nothing else!), bam!, it succeeds:

time="2017-04-21T20:21:30Z" level=info msg="requesting certificate for n.alp.im" context="ingress_tls" name=myngx namespace=default
time="2017-04-21T20:21:34Z" level=info msg="authorization successful" context=acme domain=n.alp.im
time="2017-04-21T20:21:35Z" level=info msg="successfully got certificate: domains=[n.alp.im] url=https://acme-staging.api.letsencrypt.org/acme/cert/faa98539be6a0f591589261efe3c84022003" context=acme
@ahmetb

This comment has been minimized.

Copy link
Author

ahmetb commented Apr 21, 2017

I enabled LEGO_LOG_LEVEL=debug and seeing this:

time="2017-04-21T21:26:04Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=echoserver namespace=default
time="2017-04-21T21:26:04Z" level=info msg="requesting certificate for n.alp.im" context="ingress_tls" name=echoserver namespace=default
time="2017-04-21T21:26:05Z" level=debug msg="testing reachablity of http://n.alp.im/.well-known/acme-challenge/_selftest" context=acme domain=n.alp.im
time="2017-04-21T21:26:06Z" level=debug msg="error while authorizing: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im
time="2017-04-21T21:26:06Z" level=debug msg="testing reachablity of http://n.alp.im/.well-known/acme-challenge/_selftest" context=acme domain=n.alp.im
time="2017-04-21T21:26:07Z" level=debug msg="error while authorizing: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im
time="2017-04-21T21:26:07Z" level=debug msg="testing reachablity of http://n.alp.im/.well-known/acme-challenge/_selftest" context=acme domain=n.alp.im
time="2017-04-21T21:26:08Z" level=debug msg="error while authorizing: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im
time="2017-04-21T21:26:09Z" level=debug msg="testing reachablity of http://n.alp.im/.well-known/acme-challenge/_selftest" context=acme domain=n.alp.im
time="2017-04-21T21:26:09Z" level=debug msg="error while authorizing: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im
time="2017-04-21T21:26:11Z" level=debug msg="testing reachablity of http://n.alp.im/.well-known/acme-challenge/_selftest" context=acme domain=n.alp.im
time="2017-04-21T21:26:11Z" level=debug msg="error while authorizing: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im
time="2017-04-21T21:26:15Z" level=debug msg="testing reachablity of http://n.alp.im/.well-known/acme-challenge/_selftest" context=acme domain=n.alp.im
time="2017-04-21T21:26:15Z" level=debug msg="error while authorizing: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im
time="2017-04-21T21:26:20Z" level=debug msg="testing reachablity of http://n.alp.im/.well-known/acme-challenge/_selftest" context=acme domain=n.alp.im
time="2017-04-21T21:26:20Z" level=debug msg="error while authorizing: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im
time="2017-04-21T21:26:26Z" level=debug msg="testing reachablity of http://n.alp.im/.well-known/acme-challenge/_selftest" context=acme domain=n.alp.im
time="2017-04-21T21:26:26Z" level=debug msg="error while authorizing: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im
time="2017-04-21T21:26:38Z" level=debug msg="testing reachablity of http://n.alp.im/.well-known/acme-challenge/_selftest" context=acme domain=n.alp.im
time="2017-04-21T21:26:38Z" level=debug msg="error while authorizing: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im
time="2017-04-21T21:26:50Z" level=debug msg="testing reachablity of http://n.alp.im/.well-known/acme-challenge/_selftest" context=acme domain=n.alp.im
time="2017-04-21T21:26:51Z" level=debug msg="error while authorizing: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im
time="2017-04-21T21:27:01Z" level=debug msg="testing reachablity of http://n.alp.im/.well-known/acme-challenge/_selftest" context=acme domain=n.alp.im
time="2017-04-21T21:27:01Z" level=debug msg="error while authorizing: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im
time="2017-04-21T21:27:38Z" level=warning msg="authorization failed after 1m0s: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im

[repeats...]

Weird enough, this http://n.alp.im/.well-known/acme-challenge/_selftest thing works just fine on my machine and on a separate container:

curl -iv  http://n.alp.im/.well-known/acme-challenge/_selftest
* Hostname was NOT found in DNS cache
*   Trying 35.186.213.175...
* Connected to n.alp.im (35.186.213.175) port 80 (#0)
> GET /.well-known/acme-challenge/_selftest HTTP/1.1
> User-Agent: curl/7.38.0
> Host: n.alp.im
> Accept: */*
>

HTTP/1.1 200 OK
Content-Type: text/plain
Date: Fri, 21 Apr 2017 21:32:27 GMT
Content-Length: 16
Via: 1.1 google

<
* Connection #0 to host n.alp.im left intact
uSHdnjZytkgQUJZO

I have no idea why kube-lego container sees a 403 from this _selftest. (I am not sure if 403 comes from _selftest either.)

@ankon

This comment has been minimized.

Copy link
Contributor

ankon commented Apr 21, 2017

Kube-Lego keeps some LE account data in a secret, and when switching the end points I needed to reset (aka remove) that secret.

@cpu

This comment has been minimized.

Copy link

cpu commented Apr 22, 2017

No registration exists matching provided key

This seems like a case where your client is using an account key that was registered with the staging environment with the production environment. Accounts are not portable across Let's Encrypt staging & production, they are entirely separate systems.

@ahmetb

This comment has been minimized.

Copy link
Author

ahmetb commented Apr 22, 2017

@cpu I am not aware what an account key is. Is it my email address? Is it some secret resource that I should delete? Since I'm a user of this project and not low level LE concepts, I have no idea how to fix this problem. Any pointers are appreciated.

@cpu

This comment has been minimized.

Copy link

cpu commented Apr 22, 2017

@ahmetb I'm sorry I can't help more directly. I'm coming at the problem from the other end with no experience with the project and only understanding of the low level concepts. I hope a maintainer/user can help you figure out a concrete solution.

@ahmetb

This comment has been minimized.

Copy link
Author

ahmetb commented Apr 23, 2017

OK I figured this out.

Turns out when you start a kube-lego container once, it creates a secret kube-lego-account in kube-lego namespace and it hardcodes the LE staging endpoint (and maybe some other stuff). Once you switch to from stage &rarr prod (or vice-versa) you have to delete this secret first and then make the KUBE_LEGO_URL change.

Keeping this issue open as it's not documented anywhere and figuring this out requires a somewhat non-beginner understanding of how things work (which, it shouldn't). Ideally if switching between LE endpoints requires deleting the secret/kube-lego-account, it should be documented. But I think the code can be refactored in a way that it replaces the saved secret based on the endpoint configured.

@ankon

This comment has been minimized.

Copy link
Contributor

ankon commented Apr 24, 2017

Indeed.

Note also #125 - pretty much the same issue.

@ahmetb

This comment has been minimized.

Copy link
Author

ahmetb commented Apr 24, 2017

@ankon thanks. Also #43... I was going to open an issue to log the endpoint it uses, apparently it's already there.

@etoews

This comment has been minimized.

Copy link

etoews commented Jul 17, 2017

Looks like it's now documented at https://github.com/jetstack/kube-lego#run-kube-lego

If you change the LEGO_URL, it is required that you delete the existing secret kube-lego-account and all certificates you want to request from the new URL.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment