Certbot performs challenge even when reusing valid authz

[mnordhoff]

My operating system is (include version):

Ubuntu 16.04 (x86-64)

I installed Certbot with (certbot-auto, OS package manager, pip, etc):

certbot-auto

I ran this command and it produced this output:

$ sudo -H certbot certonly --staging --webroot -w /srv/www/certbot -d 2017-12-20.clover.letsencrypt.mattnordhoff.net
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for 2017-12-20.clover.letsencrypt.mattnordhoff.net
Using the webroot path /srv/www/certbot for all unmatched domains.
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/2017-12-20.clover.letsencrypt.mattnordhoff.net/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/2017-12-20.clover.letsencrypt.mattnordhoff.net/privkey.pem
   Your cert will expire on 2018-03-20. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"

$ sudo -H certbot delete --cert-name 2017-12-20.clover.letsencrypt.mattnordhoff.net
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Deleted all files relating to certificate
2017-12-20.clover.letsencrypt.mattnordhoff.net.
-------------------------------------------------------------------------------

$ sudo -H certbot certonly --staging --webroot -w /srv/www/certbot -d 2017-12-20.clover.letsencrypt.mattnordhoff.net
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for 2017-12-20.clover.letsencrypt.mattnordhoff.net
Using the webroot path /srv/www/certbot for all unmatched domains.
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/2017-12-20.clover.letsencrypt.mattnordhoff.net/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/2017-12-20.clover.letsencrypt.mattnordhoff.net/privkey.pem
   Your cert will expire on 2018-03-20. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"

Certbot's behavior differed from what I expected because:

The second time, even though Let's Encrypt returned a valid authz, Certbot still created the challenge file and POSTed to the /acme/challenge endpoint despite it being unnecessary.

It's harmless for automated usage, but unfortunate for people using manual mode and suffering through a cumbersome file upload or DNS update process.

Here is a Certbot log showing the issue (if available):

Logs are stored in /var/log/letsencrypt by default. Feel free to redact domains, e-mail and IP addresses as you see fit.

• First run: https://mn0.us/CgfUq946UzLG9VUFKRMes5t/letsencrypt.log
• Second run: https://mn0.us/Niwqr8QLE_dULTHfVtKccxN/letsencrypt.log

Here is the relevant nginx server block or Apache virtualhost for the domain I am configuring:

[SwartzCr]

@bmw thoughts?

[bmw]

I'm actually a little surprised we didn't already have an issue about this. Thanks a lot for opening one.

Much of Certbot's core ACME functionality hasn't changed in a long time (with the exception of ongoing work now for ACMEv2) and instead we've been focusing on building more stuff on top of it. When the relevant code here was written, no one was talking about authz reuse and the new-authz endpoint always returned a new authz with pending challenges.

I agree it's usually harmless, but is an unfortunate burden for people doing things manually. We probably won't get to this in the next couple months and we're focusing on other things (like getting our automated DNS plugins packaged), but we certainly want to fix this at some point. If someone is interested in writing a PR, the relevant code here is in certbot/auth_handler.py.