certbot-auto renew --renew-hook didn't run on successful renewal #5034
My operating system is (include version):
Ubuntu 14.04.5 LTS
I installed Certbot with (certbot-auto, OS package manager, pip, etc):
I ran this command and it produced this output:
root@superunknown:/opt/cerbot# ls -al /opt/cerbot/berzerk-renew-hook.sh
The following certs are not due for renewal yet:
new certificate deployed without reload, fullchain is
Congratulations, all renewals succeeded. The following certs have been renewed:
Certbot's behavior differed from what I expected because:
I expected my hook to be run but it did not. My hook script is as follows:
root@superunknown:/opt/cerbot# cat berzerk-renew-hook.sh
echo "renew hook script"
# combine pems for courier
# restart courier
# restart postfix
Here is a Certbot log showing the issue (if available):
Logs are stored in
When you run the renewal you either need to specify the renew hook in your command, such as
I'm having exactly the same problem, the renew hook has never run in the three times that renewal has been done since I started using letsencrypt. A frustrating thing about this is it's next to impossible to debug - renewal hooks don't run during dryruns, and once the renewal happens, you have to wait another 2 months before testing it again. There is no output in
Here's some filtered output from the logs:
And here's what's in
One thing that I notice, the logs say that arguments is
Ok, I've worked out what the problem is. It seems the Ubuntu certbot (installed following the instructions for Ubuntu 16.04 with nginx here) installs both a cron job, and a systemctl timer. Why it does both I don't understand at all. The cronjob was never actually running, because it runs as the certbot user, and /var/log/letsencrypt was owned by root. The systemctl timer was running, but wasn't configured to run the pre/post/renewal hooks. There seems to be something badly messed up here with the Ubuntu packages, one or the other method of running a scheduled job should be installed. The systemctl timer doesn't even seem to be documented, how is anyone meant to know its there and how to update it?
I'm using Ubuntu 16.04. Certbot 0.19.0. I think the biggest thing that's needed here is documentation. The Ubuntu package installs a systemd timer (I'm 100% sure of that because I didn't know systemd timers existed until I happened across a post that mentioned a cerbot systemd timer, I think the script may have been there but I had to modify it to add the renew hook arguments). I'm also reasonably confident that the package also installed a cron job in
Furthermore, the certbot documentation on renewals, here, makes no mention of systemd, only cron.
So I would say the following would go along way to solving this issue:
Furthermore you should probably check if I'm right that the Ubuntu certbot packages are installing both a cron job and a systemd timer, because if it is both, that's a bug.
A final thing that would help I think is example hooks along with the installation. It's currently really hard to know what the "right" way to install hooks is, the documentation both mentions
That all makes sense, and these should be documented.
referenced this issue
Jan 17, 2018
Reading what @jroper found gave me some head ache. I had no idea that certbot installed a systemd timer on my system (debian). But i checked, and it did.
I believe this could explain why my renew-hook isn't working. I always believed that I had to install a cron job in order to automatically renew my certificates and restart my webserver with a renew-hook. As of now, I have some suspicions that my cron job never renews my certificates (because the systemd service takes care of it) and therefore never restarts my web server, which would be a big problem.
Please update the documentation so that we can understand how this works! Currently, I can't find "renew-hook" being mentioned at all in the documentation. Is the renew-hook deprecated in favor of pre- and post-hook?
I have experienced a similar problem recently where one or more versions of Certbot have installed both a cronjob and a SystemD timer on Ubuntu 16.04. The cronjob contained the "post-hook" command to reload nginx, but the LetsEncrypt config file used by the "certbot" script when called from the SystemD timer did not. I didn't know about the timer until finding this issue.
So I had a situation where the SystemD timer ran after the expiry, and successfully renewed the certificate (its webroot is described in the config file) but didn't reload nginx. It seems that the cronjob checks for the existence of the SystemD, and won't actually run if the systemd directory exists:
I should probably have paid more attention to the full command in
My workaround was to comment out the cronjob (it doesn't ever run anyway if the SystemD services are installed; see the
In version 0.21.1 of Certbot, the command-line option is
And when you run this it adds the following line to the Certbot config file (in the
So in short, a single manual forced renewal was enough to update the config file and "fix" all future executions of Certbot.
I found that a good way to test whether Certbot will run the deploy-hook/renew-hook is to do a
If you don't see the
I hope that helps someone!
also on ubuntu 16.04 with certbot 0.21 , and i think that the files installed with the ubuntu package make sense
The package also installs two systemd files a certbot.timer and a certbot.service.
Which is why the configuration files
pre-hook = /bin/run-parts /path/to/hooks/pre/ post-hook = /bin/run-parts /path/to/hooks/post/ renew-hook = /bin/run-parts /path/to/hooks/deploy/
or single commands (scripts)
pre-hook = systemctl stop nginx post-hook = systemctl start nginx renew-hook = systemctl restart nginx
if you want to go with
The conclusion here is that configuring the options in a central location like cli.ini or your-site.conf , you can run
Thanks @CliveJL for your findings - I've followed your examples and they seem to make a lot of sense!
I think the biggest issue here is that i've found countless of unofficial LE / Certbot guides out there recommending adding a custom cron-job which in light of these findings is a really bad idea.
This is very very bad from ubuntu/debian's maintainer. Why the heck provide a cronjob AND a systemd service? I was so confused now because when i executed the cronjob manually, it worked, but on the production system the certificates did not get updated. I always had to do this manually. Now i know whats going on. The cronjob is never executed.