certbot-auto renew --renew-hook didn't run on successful renewal #5034
Comments
|
When you run the renewal you either need to specify the renew hook in your command, such as |
|
as you can see above, my command does include --renew-hook and a full path to the shell script. please review the logs above |
|
Sorry! The markdown formatting made it hard for me to read your log. |
|
There isn't anything of value in those logs, I will keep an eye on it and see during my next renewal window. |
|
I'm having exactly the same problem, the renew hook has never run in the three times that renewal has been done since I started using letsencrypt. A frustrating thing about this is it's next to impossible to debug - renewal hooks don't run during dryruns, and once the renewal happens, you have to wait another 2 months before testing it again. There is no output in Here's some filtered output from the logs: And here's what's in One thing that I notice, the logs say that arguments is |
|
Ok, I've worked out what the problem is. It seems the Ubuntu certbot (installed following the instructions for Ubuntu 16.04 with nginx here) installs both a cron job, and a systemctl timer. Why it does both I don't understand at all. The cronjob was never actually running, because it runs as the certbot user, and /var/log/letsencrypt was owned by root. The systemctl timer was running, but wasn't configured to run the pre/post/renewal hooks. There seems to be something badly messed up here with the Ubuntu packages, one or the other method of running a scheduled job should be installed. The systemctl timer doesn't even seem to be documented, how is anyone meant to know its there and how to update it? |
|
@jroper I'm really sorry about that - can you let me know what version of Ubuntu/certbot you're on? |
|
I'm using Ubuntu 16.04. Certbot 0.19.0. I think the biggest thing that's needed here is documentation. The Ubuntu package installs a systemd timer (I'm 100% sure of that because I didn't know systemd timers existed until I happened across a post that mentioned a cerbot systemd timer, I think the script may have been there but I had to modify it to add the renew hook arguments). I'm also reasonably confident that the package also installed a cron job in
Furthermore, the certbot documentation on renewals, here, makes no mention of systemd, only cron. So I would say the following would go along way to solving this issue:
Furthermore you should probably check if I'm right that the Ubuntu certbot packages are installing both a cron job and a systemd timer, because if it is both, that's a bug. A final thing that would help I think is example hooks along with the installation. It's currently really hard to know what the "right" way to install hooks is, the documentation both mentions |
|
That all makes sense, and these should be documented. |
|
Just fyi that there is a PR for this #5460 and a corresponding issue for the website here, hopefully I'll have a PR done by later today, and have the website updated this week, certbot/website#305 |
|
Reading what @jroper found gave me some head ache. I had no idea that certbot installed a systemd timer on my system (debian). But i checked, and it did. I believe this could explain why my renew-hook isn't working. I always believed that I had to install a cron job in order to automatically renew my certificates and restart my webserver with a renew-hook. As of now, I have some suspicions that my cron job never renews my certificates (because the systemd service takes care of it) and therefore never restarts my web server, which would be a big problem. Please update the documentation so that we can understand how this works! Currently, I can't find "renew-hook" being mentioned at all in the documentation. Is the renew-hook deprecated in favor of pre- and post-hook? |
|
I have experienced a similar problem recently where one or more versions of Certbot have installed both a cronjob and a SystemD timer on Ubuntu 16.04. The cronjob contained the "post-hook" command to reload nginx, but the LetsEncrypt config file used by the "certbot" script when called from the SystemD timer did not. I didn't know about the timer until finding this issue. So I had a situation where the SystemD timer ran after the expiry, and successfully renewed the certificate (its webroot is described in the config file) but didn't reload nginx. It seems that the cronjob checks for the existence of the SystemD, and won't actually run if the systemd directory exists:
I should probably have paid more attention to the full command in |
|
So how did you cope with this problem, @CliveJL ? |
|
So, as I understand it, best practice for using hooks is to modify the command in the Systemd timer? Can someone confirm this? |
|
My workaround was to comment out the cronjob (it doesn't ever run anyway if the SystemD services are installed; see the In version 0.21.1 of Certbot, the command-line option is
And when you run this it adds the following line to the Certbot config file (in the
So in short, a single manual forced renewal was enough to update the config file and "fix" all future executions of Certbot. I found that a good way to test whether Certbot will run the deploy-hook/renew-hook is to do a Command: Output contains: If you don't see the I hope that helps someone! |
|
Thanks @CliveJL, that is a huge help - I had no idea why my renew-hook in my crontab was not running. Your solution is fantastic and appears to work perfectly (well, I wont know for sure for a couple months, but it looks exactly right and explains the symptoms). |
|
also on ubuntu 16.04 with certbot 0.21 , and i think that the files installed with the ubuntu package make sense The package also installs two systemd files a certbot.timer and a certbot.service. Which is why the configuration files pre-hook = /bin/run-parts /path/to/hooks/pre/
post-hook = /bin/run-parts /path/to/hooks/post/
renew-hook = /bin/run-parts /path/to/hooks/deploy/or single commands (scripts) pre-hook = systemctl stop nginx
post-hook = systemctl start nginx
renew-hook = systemctl restart nginxif you want to go with The conclusion here is that configuring the options in a central location like cli.ini or your-site.conf , you can run |
|
Thanks @CliveJL for your findings - I've followed your examples and they seem to make a lot of sense! I think the biggest issue here is that i've found countless of unofficial LE / Certbot guides out there recommending adding a custom cron-job which in light of these findings is a really bad idea. |
|
This is very very bad from ubuntu/debian's maintainer. Why the heck provide a cronjob AND a systemd service? I was so confused now because when i executed the cronjob manually, it worked, but on the production system the certificates did not get updated. I always had to do this manually. Now i know whats going on. The cronjob is never executed. |
My operating system is (include version):
Ubuntu 14.04.5 LTS
Kernel 4.9.15-x86_64
I installed Certbot with (certbot-auto, OS package manager, pip, etc):
git clone
current version: 0.17.0
I ran this command and it produced this output:
root@superunknown:/opt/cerbot# ls -al /opt/cerbot/berzerk-renew-hook.sh
-rwxr-xr-x 1 root root 300 May 19 11:27 /opt/cerbot/berzerk-renew-hook.sh
cd /opt/cerbot
./certbot-auto renew --renew-hook /opt/cerbot/berzerk-renew-hook.sh
Processing /etc/letsencrypt/renewal/mail.berzerk.org.conf
The following certs are not due for renewal yet:
/etc/letsencrypt/live/mail.berzerk.org/fullchain.pem (skipped)
No renewals were attempted.
Not Before: May 19 15:10:00 2017 GMT
Not After : Aug 17 15:10:00 2017 GMT
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mail.berzerk.org
Waiting for verification...
Cleaning up challenges
Processing /etc/letsencrypt/renewal/mail.berzerk.org.conf
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/mail.berzerk.org/fullchain.pem
Congratulations, all renewals succeeded. The following certs have been renewed:
/etc/letsencrypt/live/mail.berzerk.org/fullchain.pem (success)
Not Before: Jul 24 06:30:00 2017 GMT
Not After : Oct 22 06:30:00 2017 GMT
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Certbot's behavior differed from what I expected because:
I expected my hook to be run but it did not. My hook script is as follows:
root@superunknown:/opt/cerbot# cat berzerk-renew-hook.sh
!/bin/bash
echo "renew hook script"
# combine pems for courier
cat /etc/letsencrypt/live/mail.berzerk.org/privkey.pem /etc/letsencrypt/live/mail.berzerk.org/fullchain.pem > /etc/courier/letsencrypt.pem
# restart courier
service courier-imap-ssl restart
# restart postfix
service postfix restart
Here is a Certbot log showing the issue (if available):
Logs are stored in
/var/log/letsencryptby default. Feel free to redact domains, e-mail and IP addresses as you see fit.Here is the relevant nginx server block or Apache virtualhost for the domain I am configuring:
The text was updated successfully, but these errors were encountered: