Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

certbot-auto renew --renew-hook didn't run on successful renewal #5034

Closed
twiz718 opened this issue Aug 17, 2017 · 19 comments

Comments

Projects
None yet
9 participants
@twiz718
Copy link

commented Aug 17, 2017

My operating system is (include version):

Ubuntu 14.04.5 LTS
Kernel 4.9.15-x86_64

I installed Certbot with (certbot-auto, OS package manager, pip, etc):

git clone
current version: 0.17.0

I ran this command and it produced this output:

root@superunknown:/opt/cerbot# ls -al /opt/cerbot/berzerk-renew-hook.sh
-rwxr-xr-x 1 root root 300 May 19 11:27 /opt/cerbot/berzerk-renew-hook.sh

cd /opt/cerbot
./certbot-auto renew --renew-hook /opt/cerbot/berzerk-renew-hook.sh

Processing /etc/letsencrypt/renewal/mail.berzerk.org.conf

The following certs are not due for renewal yet:
/etc/letsencrypt/live/mail.berzerk.org/fullchain.pem (skipped)
No renewals were attempted.
Not Before: May 19 15:10:00 2017 GMT
Not After : Aug 17 15:10:00 2017 GMT
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mail.berzerk.org
Waiting for verification...
Cleaning up challenges

Processing /etc/letsencrypt/renewal/mail.berzerk.org.conf

new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/mail.berzerk.org/fullchain.pem

Congratulations, all renewals succeeded. The following certs have been renewed:
/etc/letsencrypt/live/mail.berzerk.org/fullchain.pem (success)
Not Before: Jul 24 06:30:00 2017 GMT
Not After : Oct 22 06:30:00 2017 GMT
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Certbot's behavior differed from what I expected because:

I expected my hook to be run but it did not. My hook script is as follows:

root@superunknown:/opt/cerbot# cat berzerk-renew-hook.sh
!/bin/bash

echo "renew hook script"

# combine pems for courier
cat /etc/letsencrypt/live/mail.berzerk.org/privkey.pem /etc/letsencrypt/live/mail.berzerk.org/fullchain.pem > /etc/courier/letsencrypt.pem

# restart courier
service courier-imap-ssl restart

# restart postfix
service postfix restart

Here is a Certbot log showing the issue (if available):

Logs are stored in /var/log/letsencrypt by default. Feel free to redact domains, e-mail and IP addresses as you see fit.

Here is the relevant nginx server block or Apache virtualhost for the domain I am configuring:

@SwartzCr

This comment has been minimized.

Copy link
Contributor

commented Aug 23, 2017

When you run the renewal you either need to specify the renew hook in your command, such as --renew-hook myscripthere.sh or have it referenced in your renewal config (which would happen if you specified it as a hook when you created the cert)
You can learn more about hooks here: https://certbot.eff.org/docs/using.html#renewing-certificates
Let me know if that doesn't work!

@twiz718

This comment has been minimized.

Copy link
Author

commented Aug 23, 2017

as you can see above, my command does include --renew-hook and a full path to the shell script. please review the logs above

@SwartzCr

This comment has been minimized.

Copy link
Contributor

commented Aug 23, 2017

Sorry! The markdown formatting made it hard for me to read your log.
Can you share the log for this with me? It should be in /var/log/letsencrypt - that'll help me know where and how certbot failed to call this renewal script.

@twiz718

This comment has been minimized.

Copy link
Author

commented Aug 24, 2017

There isn't anything of value in those logs, I will keep an eye on it and see during my next renewal window.

@twiz718 twiz718 closed this Aug 24, 2017

@jroper

This comment has been minimized.

Copy link

commented Dec 20, 2017

I'm having exactly the same problem, the renew hook has never run in the three times that renewal has been done since I started using letsencrypt. A frustrating thing about this is it's next to impossible to debug - renewal hooks don't run during dryruns, and once the renewal happens, you have to wait another 2 months before testing it again. There is no output in /var/log/letsencrypt that makes any mention of renewal hooks - I'm not sure that I should paste it's output here because it contains a lot of tokens that look like they shouldn't be put in a public forum.

Here's some filtered output from the logs:

2017-12-16 13:44:55,030:DEBUG:certbot.main:certbot version: 0.19.0
2017-12-16 13:44:55,030:DEBUG:certbot.main:Arguments: ['-q']
2017-12-16 13:44:55,030:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2017-12-16 13:44:55,050:DEBUG:certbot.log:Root logging level set at 30
2017-12-16 13:44:55,050:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2017-12-16 13:44:55,060:DEBUG:certbot.plugins.selection:Requested authenticator <certbot.cli._Default object at 0x7f9e593c9390> and installer <certbot.cli._Default object at 0x7f9e593c9390>
2017-12-16 13:44:55,060:DEBUG:certbot.cli:Default Detector is Namespace(account=<certbot.cli._Default object at 0x7f9e59428110>, agree_dev_preview=None, allow_subset_of_names=<certbot.cli._Default object at 0x7f9e59402390>, apache=<certbot.cli._Default object at 0x7f9e593c9690>, authenticator=<certbot.cli._Default object at 0x7f9e593c9390>, break_my_certs=<certbot.cli._Default object at 0x7f9e594190d0>, cert_path=<certbot.cli._Default object at 0x7f9e593c5b50>, certname=<certbot.cli._Default object at 0x7f9e59412510>, chain_path=<certbot.cli._Default object at 0x7f9e593c5e50>, checkpoints=<certbot.cli._Default object at 0x7f9e593c5650>, config_dir=<certbot.cli._Default object at 0x7f9e593c5f50>, config_file=None, configurator=<certbot.cli._Default object at 0x7f9e593c9390>, csr=<certbot.cli._Default object at 0x7f9e593c5450>, debug=<certbot.cli._Default object at 0x7f9e59402ad0>, debug_challenges=<certbot.cli._Default object at 0x7f9e59402dd0>, deploy_hook=<certbot.cli._Default object at 0x7f9e59428c90>, dialog=None, directory_hooks=<certbot.cli._Default object at 0x7f9e59428e90>, dns_cloudflare=<certbot.cli._Default object at 0x7f9e593c9b90>, dns_cloudxns=<certbot.cli._Default object at 0x7f9e593c9c90>, dns_digitalocean=<certbot.cli._Default object at 0x7f9e593c9d90>, dns_dnsimple=<certbot.cli._Default object at 0x7f9e593c9e90>, dns_dnsmadeeasy=<certbot.cli._Default object at 0x7f9e593c9f90>, dns_google=<certbot.cli._Default object at 0x7f9e593ce0d0>, dns_luadns=<certbot.cli._Default object at 0x7f9e593ce1d0>, dns_nsone=<certbot.cli._Default object at 0x7f9e593ce2d0>, dns_rfc2136=<certbot.cli._Default object at 0x7f9e593ce3d0>, dns_route53=<certbot.cli._Default object at 0x7f9e593ce4d0>, domains=<certbot.cli._Default object at 0x7f9e59412690>, dry_run=<certbot.cli._Default object at 0x7f9e59412390>, duplicate=<certbot.cli._Default object at 0x7f9e59428310>, eff_email=<certbot.cli._Default object at 0x7f9e59402e10>, email=<certbot.cli._Default object at 0x7f9e59402f90>, expand=<certbot.cli._Default object at 0x7f9e59402990>, force_interactive=<certbot.cli._Default object at 0x7f9e59412810>, fullchain_path=<certbot.cli._Default object at 0x7f9e593c5d50>, func=<function renew at 0x7f9e5f72a7d0>, hsts=<certbot.cli._Default object at 0x7f9e59421a10>, http01_address=<certbot.cli._Default object at 0x7f9e59412d90>, http01_port=<certbot.cli._Default object at 0x7f9e59412a90>, ifaces=<certbot.cli._Default object at 0x7f9e593c5950>, init=<certbot.cli._Default object at 0x7f9e593c5750>, installer=<certbot.cli._Default object at 0x7f9e593c9390>, key_path=<certbot.cli._Default object at 0x7f9e593c5c50>, logs_dir=<certbot.cli._Default object at 0x7f9e593c9190>, manual=<certbot.cli._Default object at 0x7f9e593c9990>, manual_auth_hook=<certbot.cli._Default object at 0x7f9e593ce610>, manual_cleanup_hook=<certbot.cli._Default object at 0x7f9e593ce750>, manual_public_ip_logging_ok=<certbot.cli._Default object at 0x7f9e593ce850>, max_log_backups=<certbot.cli._Default object at 0x7f9e59412b10>, must_staple=<certbot.cli._Default object at 0x7f9e59421210>, nginx=<certbot.cli._Default object at 0x7f9e593c9790>, no_bootstrap=<certbot.cli._Default object at 0x7f9e59402290>, no_self_upgrade=<certbot.cli._Default object at 0x7f9e59428510>, no_verify_ssl=<certbot.cli._Default object at 0x7f9e59412190>, noninteractive_mode=<certbot.cli._Default object at 0x7f9e59412990>, num=<certbot.cli._Default object at 0x7f9e593c5150>, os_packages_only=<certbot.cli._Default object at 0x7f9e59428410>, post_hook=<certbot.cli._Default object at 0x7f9e59428a90>, pre_hook=<certbot.cli._Default object at 0x7f9e59428990>, pref_challs=<certbot.cli._Default object at 0x7f9e59428890>, prepare=<certbot.cli._Default object at 0x7f9e593c5850>, quiet=True, reason=<certbot.cli._Default object at 0x7f9e593c5550>, redirect=<certbot.cli._Default object at 0x7f9e594214d0>, register_unsafely_without_email=<certbot.cli._Default object at 0x7f9e59412210>, reinstall=<certbot.cli._Default object at 0x7f9e59402b10>, renew_by_default=<certbot.cli._Default object at 0x7f9e59402650>, renew_hook=<certbot.cli._Default object at 0x7f9e59428b90>, renew_with_new_domains=<certbot.cli._Default object at 0x7f9e59402190>, rsa_key_size=<certbot.cli._Default object at 0x7f9e594210d0>, server=<certbot.cli._Default object at 0x7f9e593c9290>, staging=<certbot.cli._Default object at 0x7f9e594027d0>, standalone=<certbot.cli._Default object at 0x7f9e593c9890>, standalone_supported_challenges=<certbot.cli._Default object at 0x7f9e593ce990>, staple=<certbot.cli._Default object at 0x7f9e594281d0>, strict_permissions=<certbot.cli._Default object at 0x7f9e59428790>, text_mode=<certbot.cli._Default object at 0x7f9e59412c90>, tls_sni_01_address=<certbot.cli._Default object at 0x7f9e59412790>, tls_sni_01_port=<certbot.cli._Default object at 0x7f9e59412490>, tos=<certbot.cli._Default object at 0x7f9e594022d0>, uir=<certbot.cli._Default object at 0x7f9e59421e90>, update_registration=<certbot.cli._Default object at 0x7f9e59412050>, user_agent=<certbot.cli._Default object at 0x7f9e593c5250>, user_agent_comment=<certbot.cli._Default object at 0x7f9e593c5350>, validate_hooks=<certbot.cli._Default object at 0x7f9e59428d90>, verb='renew', verbose_count=<certbot.cli._Default object at 0x7f9e59412e10>, webroot=<certbot.cli._Default object at 0x7f9e593c9a90>, webroot_map=<certbot.cli._Default object at 0x7f9e593ceb90>, webroot_path=<certbot.cli._Default object at 0x7f9e593ce5d0>, work_dir=<certbot.cli._Default object at 0x7f9e593c9090>)
2017-12-16 13:44:55,073:DEBUG:certbot.storage:Should renew, less than 30 days before certificate expiry 2018-01-15 12:55:47 UTC.
2017-12-16 13:44:55,073:INFO:certbot.renewal:Cert is due for renewal, auto-renewing...
2017-12-16 13:44:55,073:DEBUG:certbot.plugins.selection:Requested authenticator webroot and installer None
2017-12-16 13:44:55,074:DEBUG:certbot.plugins.selection:Single candidate plugin: * webroot
Description: Place files in webroot directory
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = certbot.plugins.webroot:Authenticator
Initialized: <certbot.plugins.webroot.Authenticator object at 0x7f9e594028d0>
Prep: True
[snip]
2017-12-16 13:45:00,368:DEBUG:certbot.storage:Writing new private key to /etc/letsencrypt/archive/*redacted*/privkey5.pem.
2017-12-16 13:45:00,368:DEBUG:certbot.storage:Writing certificate to /etc/letsencrypt/archive/*redacted*/cert5.pem.
2017-12-16 13:45:00,368:DEBUG:certbot.storage:Writing chain to /etc/letsencrypt/archive/*redacted*/chain5.pem.
2017-12-16 13:45:00,369:DEBUG:certbot.storage:Writing full chain to /etc/letsencrypt/archive/*redacted*/fullchain5.pem.
2017-12-16 13:45:00,389:DEBUG:certbot.storage:Writing new config /etc/letsencrypt/renewal/*redacted*.conf.new.
2017-12-16 13:45:00,392:DEBUG:certbot.renewal:no renewal failures

And here's what's in /etc/cron.d/certbot:

0 */12 * * * certbot test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(3600))' && certbot -q renew --pre-hook '/bin/run-parts /etc/letsencrypt/pre-hook.d/' --post-hook '/bin/run-parts /etc/letsencrypt/post-hook.d/' --renew-hook '/bin/run-parts /etc/letsencrypt/renew-hook.d/'

One thing that I notice, the logs say that arguments is -q, there's no mention of my hook arguments.

@jroper

This comment has been minimized.

Copy link

commented Dec 20, 2017

Ok, I've worked out what the problem is. It seems the Ubuntu certbot (installed following the instructions for Ubuntu 16.04 with nginx here) installs both a cron job, and a systemctl timer. Why it does both I don't understand at all. The cronjob was never actually running, because it runs as the certbot user, and /var/log/letsencrypt was owned by root. The systemctl timer was running, but wasn't configured to run the pre/post/renewal hooks. There seems to be something badly messed up here with the Ubuntu packages, one or the other method of running a scheduled job should be installed. The systemctl timer doesn't even seem to be documented, how is anyone meant to know its there and how to update it?

@SwartzCr

This comment has been minimized.

Copy link
Contributor

commented Dec 20, 2017

@jroper I'm really sorry about that - can you let me know what version of Ubuntu/certbot you're on?
We've had a lot of issues with renew hooks, and we're working to change them to make them more reliable and intuitive

@jroper

This comment has been minimized.

Copy link

commented Dec 20, 2017

I'm using Ubuntu 16.04. Certbot 0.19.0. I think the biggest thing that's needed here is documentation. The Ubuntu package installs a systemd timer (I'm 100% sure of that because I didn't know systemd timers existed until I happened across a post that mentioned a cerbot systemd timer, I think the script may have been there but I had to modify it to add the renew hook arguments). I'm also reasonably confident that the package also installed a cron job in /etc/cron.d, but it was a while ago that I set it up so it is possible that I put that there following some instructions - though the cron job didn't look like anything I'd write, I tend to keep things simple but it included all the logic for testing that certbot was installed etc, as you'd expect a distribution installed script to do. It's also possible that it was left there as a remnant of the certbot package switching at some point. But if you read the instructions about auto renewal for Ubuntu 16.04, it says this:

The Certbot packages on your system come with a cron job that will renew your certificates automatically before they expire.

Furthermore, the certbot documentation on renewals, here, makes no mention of systemd, only cron.

So I would say the following would go along way to solving this issue:

  1. Update the certbot installation documentation for Ubuntu to correctly document that certbot installs with a systemd timer, rather than a cron job.
  2. Update the certbot usage documentation to mention that some packages setup a systemd script.

Furthermore you should probably check if I'm right that the Ubuntu certbot packages are installing both a cron job and a systemd timer, because if it is both, that's a bug.

A final thing that would help I think is example hooks along with the installation. It's currently really hard to know what the "right" way to install hooks is, the documentation both mentions /etc/letsencrypt/renewal-hooks as well as --deploy-hook. Most people just want to restart their webserver as a hook, so including some examples for each webserver that can be copied into the renewal-hooks directory would help a lot.

@SwartzCr

This comment has been minimized.

Copy link
Contributor

commented Jan 5, 2018

That all makes sense, and these should be documented.
The reason that it installs both is that some systems run one, and some run the other. On systems where both are installed it'll default to systemd, on systems without systemd it'll use the cron job.
That said, I'm a little confused. You mention that because the cron job wasn't running your hooks weren't executed, but then you also say that you didn't write the cron job yourself.
How were you specifying your hooks? Did you manually add them to the cron job? Or did you write them as flags when initially generating your certificate?
I'd be happy to extend the documentation, though I might ask @hlieberman to write an initial draft of the text describing how timers work on Ubuntu and Debian.

@SwartzCr

This comment has been minimized.

Copy link
Contributor

commented Jan 24, 2018

Just fyi that there is a PR for this #5460 and a corresponding issue for the website here, hopefully I'll have a PR done by later today, and have the website updated this week, certbot/website#305

@victor-lund

This comment has been minimized.

Copy link

commented Feb 12, 2018

Reading what @jroper found gave me some head ache. I had no idea that certbot installed a systemd timer on my system (debian). But i checked, and it did.

I believe this could explain why my renew-hook isn't working. I always believed that I had to install a cron job in order to automatically renew my certificates and restart my webserver with a renew-hook. As of now, I have some suspicions that my cron job never renews my certificates (because the systemd service takes care of it) and therefore never restarts my web server, which would be a big problem.

Please update the documentation so that we can understand how this works! Currently, I can't find "renew-hook" being mentioned at all in the documentation. Is the renew-hook deprecated in favor of pre- and post-hook?

@CliveJL

This comment has been minimized.

Copy link

commented Feb 19, 2018

I have experienced a similar problem recently where one or more versions of Certbot have installed both a cronjob and a SystemD timer on Ubuntu 16.04. The cronjob contained the "post-hook" command to reload nginx, but the LetsEncrypt config file used by the "certbot" script when called from the SystemD timer did not. I didn't know about the timer until finding this issue.

So I had a situation where the SystemD timer ran after the expiry, and successfully renewed the certificate (its webroot is described in the config file) but didn't reload nginx. It seems that the cronjob checks for the existence of the SystemD, and won't actually run if the systemd directory exists:

test -x /usr/bin/certbot -a \! -d /run/systemd/system

I should probably have paid more attention to the full command in /etc/cron.d/certbot but I was believing that the full command line in the cronjob would be looking after my LetsEncrypt certificate for me, whereas it was in fact the SystemD timer with its incomplete config file.

@dimbula2000

This comment has been minimized.

Copy link

commented Mar 3, 2018

So how did you cope with this problem, @CliveJL ?
Stop checking systemd directory existence in cronjob script ?
It would be very helpful.

@victor-lund

This comment has been minimized.

Copy link

commented Mar 4, 2018

So, as I understand it, best practice for using hooks is to modify the command in the Systemd timer? Can someone confirm this?

@CliveJL

This comment has been minimized.

Copy link

commented Mar 5, 2018

My workaround was to comment out the cronjob (it doesn't ever run anyway if the SystemD services are installed; see the test command above), and then run a single --force-renewal providing the --deploy-hook command that you want passed in when the SystemD timer executes certbot. What I found is that if you manually run a --force-renewal then it updates the Certbot config file (/etc/letsencrypt/renewal/<sitename>) with the command you want to run after a successful certificate update and deployment.

In version 0.21.1 of Certbot, the command-line option is --deploy-hook, e.g.

certbot renew --force-renewal --deploy-hook "service nginx reload"

And when you run this it adds the following line to the Certbot config file (in the [renewalparams] section), which will be used for all future executions of Certbot by the SystemD timer:

renew_hook = service nginx reload

So in short, a single manual forced renewal was enough to update the config file and "fix" all future executions of Certbot.

I found that a good way to test whether Certbot will run the deploy-hook/renew-hook is to do a --dry-run command after updating/editing the Certbot config file with the renew_hook line above. What you should see in the --dry-run output is the following:

Command:
certbot renew --dry-run (no need to provide the --deploy-hook if you've edited the config file)

Output contains:

Waiting for verification...
Cleaning up challenges
Dry run: skipping deploy hook command: service nginx reload

If you don't see the skipping deploy hook in the dry-run output then it's not likely to be run when the SystemD timer executes certbot. As I say, the fix is to update the Certbot config file for your site(s) either by running a --force-renewal as you want it, or by just editing the file yourself.

I hope that helps someone!

@peternlewis

This comment has been minimized.

Copy link

commented Mar 8, 2018

Thanks @CliveJL, that is a huge help - I had no idea why my renew-hook in my crontab was not running. Your solution is fantastic and appears to work perfectly (well, I wont know for sure for a couple months, but it looks exactly right and explains the symptoms).

@aduzsardi

This comment has been minimized.

Copy link

commented Mar 10, 2018

also on ubuntu 16.04 with certbot 0.21 , and i think that the files installed with the ubuntu package make sense
i mean , there are variants of ubuntu/debian that don't use systemd as their init system, in that case the /etc/cron.d/cerbot will run as usual issuing "certbot -q renew" twice a day , there's no need to disabled it ... it will do nothing if you are on a ubuntu variant with systemd. (you can disable it if you really want to , just by renaming it to something like cerbot.dist or certbot.disabled ... basically anything with a dot in it)

The package also installs two systemd files a certbot.timer and a certbot.service.
certbot.timer will wake up certbot.service twice a day (just like a cronjob) which (the service) will run certbot -q renew , nothing differently than the cronjob.

Which is why the configuration files /etc/letsencrypt/cli.ini or /etc/letsencrypt/renewal/<site>.conf is the best place to add the options like hooks , or just place executable shell scripts in /etc/letsencrypt/renewal-hooks/pre/ , /etc/letsencrypt/renewal-hooks/post/ , /etc/letsencrypt/renewal-hooks/deploy/ as specified by the documentation

pre-hook = /bin/run-parts /path/to/hooks/pre/
post-hook = /bin/run-parts /path/to/hooks/post/
renew-hook = /bin/run-parts /path/to/hooks/deploy/

or single commands (scripts)

pre-hook = systemctl stop nginx
post-hook = systemctl start nginx
renew-hook = systemctl restart nginx

if you want to go with /bin/run-parts , there are some gotchas concerning the filenames (script names)
by default can contain only [a-zA-z0-9_-] , must be executable , and are run in lexical sort order.
you can set other options if you want , see man 8 run-parts

The conclusion here is that configuring the options in a central location like cli.ini or your-site.conf , you can run certbot renew from anywhere with the same options.

@victor-lund

This comment has been minimized.

Copy link

commented Mar 14, 2018

Thanks @CliveJL for your findings - I've followed your examples and they seem to make a lot of sense!

I think the biggest issue here is that i've found countless of unofficial LE / Certbot guides out there recommending adding a custom cron-job which in light of these findings is a really bad idea.

@damnms

This comment has been minimized.

Copy link

commented Feb 28, 2019

This is very very bad from ubuntu/debian's maintainer. Why the heck provide a cronjob AND a systemd service? I was so confused now because when i executed the cronjob manually, it worked, but on the production system the certificates did not get updated. I always had to do this manually. Now i know whats going on. The cronjob is never executed.
Thanks @CliveJL

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.