systemd timers not triggering

[lemmy04]

I have a matrix homeserver installed with this play, the systemd timers installed by it did not fire.
See this screenshot: https://paste.pics/GVQKL

To Reproduce
My vars.yml file looks like this with the sensitive stuff stripped out:

matrix_domain: eregion.de
matrix_ssl_lets_encrypt_support_email: 'admin@eregion.de'
matrix_nginx_proxy_base_domain_serving_enabled: false
matrix_coturn_turn_external_ip_address: 89.58.34.47
matrix_mailer_sender_address: "admin@eregion.de"
matrix_mailer_relay_use: true
matrix_mailer_relay_host_name: "*.*.de"
matrix_mailer_relay_host_port: 587
matrix_mailer_relay_auth: true
matrix_synapse_report_stats: true
matrix_synapse_admin_enabled: true
matrix_registration_enabled: true
matrix_dimension_enabled: false
matrix_jitsi_enabled: false
matrix_prometheus_enabled: true
matrix_prometheus_node_exporter_enabled: true
matrix_grafana_enabled: true
matrix_grafana_anonymous_access: false
matrix_homeserver_generic_secret_key: "{{ matrix_synapse_macaroon_secret_key }}"

Expected behavior
The letsencrypt certificate is supposed to get renewed automatically by the systemd timers.

Matrix Server:

• OS: Ubuntu 20.04.4 LTS
• Architecture X86_64

Additional context
Just a wild guess here:
I ran the play just the day before the certificates expired, to upgrade my server to the latest versions.
maybe that re-installed those timers, so they didn't run before the expiry time?

[spantaleev]

I haven't heard of others with an issue like this, so I wonder if it's got something to do with your install only.

If you were using another webserver (matrix_nginx_proxy_enabled: false), I would have suspected that the certificates were being renewed and that you just weren't restarting/reloading that other webserver. But you're using matrix-nginx-proxy, so reloading the server periodically is also done automatically on a timer.

Are you really on Ubuntu 21.04? Which architecture? It's a little bit hard to tell if you've left this section of the issue template empty.

[lemmy04]

I'm on 20.04.1 LTS X86_64.
here's the output of systemctl list.-timers, the letsencrypt renewal timer should have fired last night from what I saw in the same output yesterday, but it didn't.

root@matrix:~# systemctl list-timers
NEXT                         LEFT          LAST                         PASSED       UNIT                                             ACTIVATES                         >
Tue 2022-05-10 12:38:37 CEST 4h 35min left Tue 2022-05-10 06:28:27 CEST 1h 34min ago ua-timer.timer                                   ua-timer.service                  >
Tue 2022-05-10 14:32:46 CEST 6h left       Mon 2022-05-09 19:23:20 CEST 12h ago      fwupd-refresh.timer                              fwupd-refresh.service             >
Tue 2022-05-10 14:57:34 CEST 6h left       Tue 2022-05-10 04:02:59 CEST 4h 0min ago  apt-daily.timer                                  apt-daily.service                 >
Tue 2022-05-10 20:05:08 CEST 12h left      Tue 2022-05-10 04:02:59 CEST 4h 0min ago  motd-news.timer                                  motd-news.service                 >
Wed 2022-05-11 00:00:00 CEST 15h left      Tue 2022-05-10 00:00:05 CEST 8h ago       logrotate.timer                                  logrotate.service                 >
Wed 2022-05-11 00:00:00 CEST 15h left      Tue 2022-05-10 00:00:05 CEST 8h ago       man-db.timer                                     man-db.service                    >
Wed 2022-05-11 04:13:15 CEST 20h left      n/a                          n/a          matrix-ssl-lets-encrypt-certificates-renew.timer matrix-ssl-lets-encrypt-certificat>
Wed 2022-05-11 06:02:58 CEST 21h left      Tue 2022-05-10 06:46:43 CEST 1h 16min ago apt-daily-upgrade.timer                          apt-daily-upgrade.service         >
Wed 2022-05-11 07:23:46 CEST 23h left      Tue 2022-05-10 06:33:28 CEST 1h 29min ago matrix-coturn-reload.timer                       matrix-coturn-reload.service      >
Wed 2022-05-11 07:27:22 CEST 23h left      Tue 2022-05-10 07:17:52 CEST 45min ago    matrix-ssl-nginx-proxy-reload.timer              matrix-ssl-nginx-proxy-reload.serv>
Wed 2022-05-11 07:51:55 CEST 23h left      Tue 2022-05-10 07:51:55 CEST 11min ago    systemd-tmpfiles-clean.timer                     systemd-tmpfiles-clean.service    >
Sun 2022-05-15 03:10:40 CEST 4 days left   Sun 2022-05-08 03:10:39 CEST 2 days ago   e2scrub_all.timer                                e2scrub_all.service               >
Mon 2022-05-16 00:00:00 CEST 5 days left   Mon 2022-05-09 00:00:04 CEST 1 day 8h ago fstrim.timer                                     fstrim.service                    >

13 timers listed.

and here's /etc/os-release:

root@matrix:~# cat /etc/os-release 
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
´´´

(fixed the server details on the OP for readability)

I'm on your matrix channel as @mathias:eregion.de 

[spantaleev]

Did you figure it out?

[lemmy04]

Not yet - the current set of certificates won't renew until august o.0

But I have a strong suspicion: I think what happened was that I did a redeploy to upgrade my server less than 24 hours before the cert expired - so the renewal job didn't run in time.

[lemmy04]

the automatic renewal of certificates has worked just fine this round.
Looks like my last idea was right - I must have redeployed / refreshed my setup less than 24 hours before the last renewal, that screwed things up.