Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LetsEncrypt not auto renewing certificates #1445

Closed
borpin opened this issue Jul 8, 2020 · 27 comments
Closed

LetsEncrypt not auto renewing certificates #1445

borpin opened this issue Jul 8, 2020 · 27 comments

Comments

@borpin
Copy link

borpin commented Jul 8, 2020

Supervised Install.

I have just had a renewal notice for the certificate for the domain name I use for HA and setup using the LetsEncrypt Plugin. On the basis of getting the email, the renewal should have already happened (certbot does not wait that late).

It appears the auto-renew is not working.

When the addon is manually started, the renewal occurs.

letsencrypt log showing previous auto renewal attempt plus the renewal done when the addon was restarted.

2020-07-08 01:50:16,689:DEBUG:certbot.main:certbot version: 0.31.0
2020-07-08 01:50:16,690:DEBUG:certbot.main:Arguments: ['-q']
2020-07-08 01:50:16,690:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#dns-cloudflare,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2020-07-08 01:50:16,722:DEBUG:certbot.log:Root logging level set at 30
2020-07-08 01:50:16,723:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2020-07-08 01:50:16,734:DEBUG:certbot.renewal:no renewal failures
2020-07-08 12:52:11,104:DEBUG:certbot.main:certbot version: 0.31.0
2020-07-08 12:52:11,105:DEBUG:certbot.main:Arguments: []
2020-07-08 12:52:11,105:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#dns-cloudflare,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2020-07-08 12:52:11,138:DEBUG:certbot.log:Root logging level set at 20
2020-07-08 12:52:11,141:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2020-07-08 12:52:11,151:DEBUG:certbot.plugins.selection:Requested authenticator None and installer None
2020-07-08 12:52:11,151:DEBUG:certbot.plugins.selection:No candidate plugin
2020-07-08 12:52:11,152:DEBUG:certbot.plugins.selection:Selected authenticator None and installer None
2020-07-08 12:59:24,483:DEBUG:certbot.main:certbot version: 0.31.0
2020-07-08 12:59:24,484:DEBUG:certbot.main:Arguments: []
2020-07-08 12:59:24,485:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#dns-cloudflare,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2020-07-08 12:59:24,512:DEBUG:certbot.log:Root logging level set at 20
2020-07-08 12:59:24,513:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log

addon log - log was empty before manual start.

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] file-structure.sh: executing... 
[cont-init.d] file-structure.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[13:02:01] INFO: Selected DNS Provider: dns-cloudflare
[13:02:02] INFO: Use propagation seconds: 60
[13:02:02] INFO: Use CloudFlare token
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-cloudflare, Installer None
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
dns-01 challenge for hassio.borpin.net
Waiting 60 seconds for DNS changes to propagate
Waiting for verification...
Cleaning up challenges
IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /data/letsencrypt/live/hassio.borpin.net/fullchain.pem
   Your key file has been saved at:
   /data/letsencrypt/live/hassio.borpin.net/privkey.pem
   Your cert will expire on 2020-10-06. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.

syslog excerpts

Jul  8 01:50:15 hassio systemd[1]: Starting Certbot...
Jul  8 01:50:16 hassio systemd[1]: certbot.service: Succeeded.
Jul  8 01:50:16 hassio systemd[1]: Started Certbot.
/var/log/syslog:Jul  8 12:00:02 hassio CRON[15469]: (root) CMD (test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew)
@agners
Copy link
Member

agners commented Jul 18, 2020

From what I understand, the supervisor has no feature which starts the Let's Encrypt add-on when necessary... By default the startup mode is set as once, which basically means "on user request". If you set it to "Start on boot", then it will get started on boot but you will have to reboot the system periodically...

From what I can tell there is no "periodically startup mode" for add-ons (see also startup section in
https://developers.home-assistant.io/docs/add-ons/configuration/). I guess a cron type of startup mode would be required for this case, which need to be implemented in the supervisor project.

@borpin
Copy link
Author

borpin commented Jul 18, 2020

@agners, no reason why there couldn't be AFAICS. I think if you modify the installed certbot config to find the right config file it will work, but I have not delved any deeper as yet.

I think the addon installs certbot to the underlying OS (although I may have done that in the past), but the standard config does not find the certificate config by default.

@lambtho12
Copy link
Contributor

lambtho12 commented Aug 3, 2020

A simple (user-based) fix for this is to launch the add-on periodically to force it to renew the certificate if needed. I use the following automation to this end

# Start Let's Encrypt every night to force renewal of certificate
- alias: system_letsencrypt_renewal
  trigger:
    - platform: time
      at: '03:00:00'
  action:
    - service: hassio.addon_restart
      data:
        addon: core_letsencrypt

Note that running this every day is probably overkill as certbot will renew it as soon as it is below 30 days left, but it does not really matters.

@ludeeus , maybe we can add this small snippet to the addon documentation while we wait for a more integrated solution?

@ludeeus
Copy link
Member

ludeeus commented Aug 3, 2020

That will only solve half the issue.
All services (like Home Assistant) will still use the old certificate.

@agners
Copy link
Member

agners commented Aug 3, 2020

@ludeeus hm, I guess because the SSL certificate is already loaded at that time? Ideally, HA Core should have a reload function to just reload the SSL certificate. I guess for add-ons there is no common way to handle that, probably just restarting them is fine.

@ludeeus
Copy link
Member

ludeeus commented Aug 4, 2020

Yeah, all weservers load the certificate on startup, and that will continue to be loaded (even if the file changes) until the server restarts.

@borpin
Copy link
Author

borpin commented Aug 4, 2020

Under normal circumstances, certbot restarts apache on renewal. I wonder if the addon could create a flag/sensor that other services monitor?

With an update cycle for core as it is, and therefore regular restarts, it is less of an issue, but thinking on that, I'd have expected the addon to start and update so now surprised I saw the expiry notice. Odd.

Copy link
Member

ludeeus commented Aug 4, 2020

That would be something that should be handled by a certificate manager in the supervisor and not by this addon.

@borpin
Copy link
Author

borpin commented Aug 4, 2020

That would be something that should be handled by a certificate manager in the supervisor

Not sure what this was a reply to, but if my comment, it should, but obviously isn't.

As I said, under normal circumstances, certbot (certificate manager) does it for you.

@ludeeus
Copy link
Member

ludeeus commented Aug 4, 2020

It was in regards to this part

I wonder if the addon could create a flag/sensor that other services monitor?

@borpin
Copy link
Author

borpin commented Aug 4, 2020

So back to my point, under normal circumstances, the tool doing the certificate renewal, i.e. managing the certificates, would do the server restart (as certbot does if the right plugin is used). Because the certificate manager is actually an addon, it cannot directly control the restart, but the addon certainly could tell supervisor/HA it needs to restart.

Supervisor does not have a certificate manager AFAICS.

@ludeeus
Copy link
Member

ludeeus commented Aug 4, 2020

The addon is not aware of which services that uses the certificates.

Supervisor does not have a certificate manager AFAICS.

Correct, and when someone adds that to the supervisor it will know which service uses which certificate and can handle that.

For now, your best option (since only you know where you are using the certificates), an automation as described here #1445 (comment)
With potentially another action for restarts of the addons (and/or core) where you use the certificates.

@lambtho12
Copy link
Contributor

lambtho12 commented Aug 4, 2020

That will only solve half the issue.
All services (like Home Assistant) will still use the old certificate.

So this should do the trick instead then (using the cert_expiry integration):

# Renew certificates and restart homeassistant when cert expires in less than 30 days
- alias: system_letsencrypt_renewal
  trigger:
    - platform: time
      at: '03:00:00'
  condition:
    - condition: numeric_state
      entity_id: sensor.ssl_certificate_expiry
      below: 30
  action:
    - service: hassio.addon_restart
      data:
        addon: core_letsencrypt
    - delay: 00:05:00
    - service: homeassistant.restart

@ludeeus
Copy link
Member

ludeeus commented Aug 4, 2020

If you only use it for homeassistant, and you can have it "randomly" restart during the night yes 👍

@agners
Copy link
Member

agners commented Aug 4, 2020

@ludeeus @borpin most web servers do also support SSL certificate reloads via signals, this avoids full blown restarts. E.g. nginx supports reload which do not lead to any downtime: https://stackoverflow.com/questions/43088363/how-nginx-reload-work-why-it-is-zero-downtime. And that is also what certbot uses by default (see https://github.com/certbot/certbot/blob/09ab4aea01aaf95a2a830ad48271aa6bd11eef84/certbot-nginx/certbot_nginx/_internal/configurator.py#L1178).

@maidau
Copy link

maidau commented Aug 13, 2020

So this should do the trick instead then (using the cert_expiry integration):

# Renew certificates and restart homeassistant when cert expires in less than 30 days
- alias: system_letsencrypt_renewal
  trigger:
    - platform: time
      at: '03:00:00'
  condition:
    - condition: numeric_state
      entity_id: sensor.ssl_certificate_expiry
      below: 30
  action:
    - service: hassio.addon_restart
      data:
        addon: core_letsencrypt
    - delay: 00:05:00
    - service: homeassistant.restart

@lambtho12 please excuse me, I'm new here :-)
Which yaml file should that code be inserted? I currently have the simple once a day time based restart code placed in system.yaml and it doesn't seem to be doing a thing (I've checked the Let's Encrypt logs).

@lambtho12
Copy link
Contributor

lambtho12 commented Aug 13, 2020

@maidau This is an automation. So it should be in automations.yaml or something like that. You could also recreate it using the automation editor of the UI (under configuration/automation) instead if you prefer. See the automation documentation for more information.

@stale
Copy link

stale bot commented Sep 12, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Sep 12, 2020
@stale stale bot closed this as completed Sep 20, 2020
@asychev
Copy link

asychev commented Jan 18, 2021

The solution above will not work anymore because of home-assistant/core#42338

@agners
Copy link
Member

agners commented Jan 20, 2021

@asychev you can use a template to calculate the time until expiry:

# Renew certificates and restart homeassistant when cert expires in less than 30 days
- alias: system_letsencrypt_renewal
  trigger:
    - platform: time
      at: '03:00:00'
  condition:
    condition: template
    value_template: {{ as_timestamp(states("sensor.cert_expiry")) - as_timestamp(now()) < 30 * 24 * 60 * 60 }}
  action:
    - service: hassio.addon_restart
      data:
        addon: core_letsencrypt
    - delay: 00:05:00
    - service: homeassistant.restart

Update: Fixed syntax

@borpin
Copy link
Author

borpin commented Jan 22, 2021

So what it needs is the following;

Restart letsencrypt addon
Restart NGINX addon (new cert shows in browser padlock)
Restart HA to update sensor.

@agners - what is the right config to restart the nginx addon (and where could I look that up for future reference?)

@ronnix
Copy link

ronnix commented Feb 7, 2021

@agners @asychev FYI, this syntax for the condition worked better for me :

# Renew certificates and restart homeassistant when cert expires in less than 30 days
- alias: system_letsencrypt_renewal
  trigger:
    - platform: time
      at: '03:00:00'
  condition:
    condition: template
    value_template: "{{as_timestamp(states('sensor.cert_expiry_timestamp_HOST_PORT')) - as_timestamp(now()) < 30 * 24 * 60 * 60}}"
  action:
    - service: hassio.addon_restart
      data:
        addon: core_letsencrypt
    - delay: 00:05:00
    - service: homeassistant.restart

@medmunds
Copy link
Contributor

@borpin the nginx addon's name appears to be core_nginx_proxy. (I don't know the right place to look that up, but I found it by looking in the URL for that addon in the Supervisor page.)

Adding it to the automation's actions seems to restart it:

# ...
  action:
    - service: hassio.addon_restart
      data:
        addon: core_letsencrypt
    - delay: 00:05:00
    - service: hassio.addon_restart
      data:
        addon: core_nginx_proxy
    - service: homeassistant.restart

(BTW, trying to call hassio.addon_restart on addon: core_nginx_proxy from the Developer Tools Services panel gives the error "Failed to call service hassio/addon_restart. undefined", but does seem to restart nginx.)

@borpin
Copy link
Author

borpin commented Mar 24, 2021

My solution worked overnight - thanks to all for your help

https://community.home-assistant.io/t/lets-encrypt-add-on-not-renewing-certificates-correctly/214294

@decentropy
Copy link

decentropy commented Jun 17, 2021

So what it needs is the following;

Restart letsencrypt addon
Restart NGINX addon (new cert shows in browser padlock)
Restart HA to update sensor.

Not sure why one needs to restart NGINX core proxy.. and then restart HA.
Wouldn't restarting HA cover both?

@borpin
Copy link
Author

borpin commented Jun 18, 2021

Not sure why one needs to restart NGINX core proxy.. and then restart HA.
Wouldn't restarting HA cover both?

Because restarting HA just restarts the core not the Supervisor (I believe). For that you need to do a full host restart (which I was trying to avoid).

Try is and see 😄

@supersebbo
Copy link

Thanks for this! I was going crazy trying to work out why CertBot hadn't updated my certificates, I assumed it was copying to the wrong place, but actually just needed to restart HA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

10 participants