New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nginx fails to restart during letsencrypt renewing because it's restarted too quickly more than 5 times #1193

Closed
dpeca opened this Issue May 12, 2017 · 1 comment

Comments

Projects
None yet
1 participant
@dpeca
Copy link
Collaborator

dpeca commented May 12, 2017

This morning I found my nginx down.

Latest code from github, latest debian8.

May 12 06:35:01 eagle CRON[11773]: (admin) CMD (sudo /usr/local/vesta/bin/v-update-letsencrypt-ssl)
May 12 06:35:02 eagle systemd[1]: Stopping LSB: Stop/start nginx...
May 12 06:35:02 eagle systemd[1]: Starting LSB: Stop/start nginx...
May 12 06:35:02 eagle systemd[1]: Started LSB: Stop/start nginx.
May 12 06:35:02 eagle systemd[1]: Reloading LSB: Apache2 web server.
May 12 06:35:02 eagle apache2[12208]: Reloading web server: apache2.
May 12 06:35:02 eagle systemd[1]: Reloaded LSB: Apache2 web server.
May 12 06:35:06 eagle systemd[1]: Stopping LSB: Stop/start nginx...
May 12 06:35:07 eagle systemd[1]: Starting LSB: Stop/start nginx...
May 12 06:35:07 eagle systemd[1]: Started LSB: Stop/start nginx.
May 12 06:35:07 eagle systemd[1]: Reloading LSB: Apache2 web server.
May 12 06:35:07 eagle apache2[12431]: Reloading web server: apache2.
May 12 06:35:07 eagle systemd[1]: Reloaded LSB: Apache2 web server.
May 12 06:35:12 eagle systemd[1]: Reloading LSB: Apache2 web server.
May 12 06:35:12 eagle apache2[12719]: Reloading web server: apache2.
May 12 06:35:12 eagle systemd[1]: Reloaded LSB: Apache2 web server.
May 12 06:35:12 eagle systemd[1]: Stopping LSB: Stop/start nginx...
May 12 06:35:12 eagle systemd[1]: Starting LSB: Stop/start nginx...
May 12 06:35:12 eagle systemd[1]: Started LSB: Stop/start nginx.
May 12 06:35:13 eagle systemd[1]: Reloading LSB: Apache2 web server.
May 12 06:35:13 eagle apache2[12884]: Reloading web server: apache2.
May 12 06:35:13 eagle systemd[1]: Reloaded LSB: Apache2 web server.
May 12 06:35:13 eagle systemd[1]: Stopping LSB: Stop/start nginx...
May 12 06:35:13 eagle systemd[1]: Starting LSB: Stop/start nginx...
May 12 06:35:13 eagle systemd[1]: Started LSB: Stop/start nginx.
May 12 06:35:13 eagle systemd[1]: Reloading LSB: Apache2 web server.
May 12 06:35:13 eagle apache2[12968]: Reloading web server: apache2.
May 12 06:35:13 eagle systemd[1]: Reloaded LSB: Apache2 web server.
May 12 06:35:14 eagle systemd[1]: Stopping LSB: Stop/start nginx...
May 12 06:35:14 eagle systemd[1]: Starting LSB: Stop/start nginx...
May 12 06:35:14 eagle systemd[1]: Started LSB: Stop/start nginx.
May 12 06:35:14 eagle systemd[1]: Reloading LSB: Apache2 web server.
May 12 06:35:14 eagle apache2[13141]: Reloading web server: apache2.
May 12 06:35:14 eagle systemd[1]: Reloaded LSB: Apache2 web server.
May 12 06:35:17 eagle systemd[1]: Stopping LSB: Stop/start nginx...
May 12 06:35:17 eagle systemd[1]: Starting LSB: Stop/start nginx...
May 12 06:35:17 eagle systemd[1]: Started LSB: Stop/start nginx.
May 12 06:35:17 eagle systemd[1]: Reloading LSB: Apache2 web server.
May 12 06:35:18 eagle apache2[13319]: Reloading web server: apache2.
May 12 06:35:18 eagle systemd[1]: Reloaded LSB: Apache2 web server.
May 12 06:35:21 eagle systemd[1]: Reloading LSB: Apache2 web server.
May 12 06:35:21 eagle apache2[13594]: Reloading web server: apache2.
May 12 06:35:21 eagle systemd[1]: Reloaded LSB: Apache2 web server.
May 12 06:35:21 eagle systemd[1]: Stopping LSB: Stop/start nginx...
May 12 06:35:21 eagle systemd[1]: Starting LSB: Stop/start nginx...
May 12 06:35:21 eagle systemd[1]: Started LSB: Stop/start nginx.
May 12 06:35:21 eagle systemd[1]: Reloading LSB: Apache2 web server.
May 12 06:35:22 eagle apache2[13759]: Reloading web server: apache2.
May 12 06:35:22 eagle systemd[1]: Reloaded LSB: Apache2 web server.
May 12 06:35:22 eagle systemd[1]: Stopping LSB: Stop/start nginx...
May 12 06:35:22 eagle systemd[1]: Starting LSB: Stop/start nginx...
May 12 06:35:22 eagle systemd[1]: nginx.service start request repeated too quickly, refusing to start.
May 12 06:35:22 eagle systemd[1]: Failed to start LSB: Stop/start nginx.
May 12 06:35:22 eagle systemd[1]: Unit nginx.service entered failed state.
May 12 06:35:22 eagle systemd[1]: Starting LSB: Stop/start nginx...
May 12 06:35:22 eagle systemd[1]: nginx.service start request repeated too quickly, refusing to start.
May 12 06:35:22 eagle systemd[1]: Failed to start LSB: Stop/start nginx.

I found it's happenig because this - https://serverfault.com/questions/845471/service-start-request-repeated-too-quickly-refusing-to-start-limit

The default limit is to allow 5 restarts in a 10sec period. If a service goes over that threshold due to the Restart= config option in the service definition, it will not attempt to restart any further.

One vesta account have 2 domains with letsencrypt.

I think we should put sleep 10 in /bin/v-update-letsencrypt-ssl foreach loop.

@dpeca dpeca changed the title nginx fails to restart during letsencrypt renewing because it's restarted too quckly more than 10 times nginx fails to restart during letsencrypt renewing because it's restarted too quickly more than 5 times May 12, 2017

@dpeca

This comment has been minimized.

Copy link
Collaborator

dpeca commented May 12, 2017

I think this can be serious issue, since I found nginx down in the morning... causing all sites offline.

@dpeca dpeca closed this in 64e6fa8 May 15, 2017

dpeca added a commit that referenced this issue Oct 18, 2017

dpeca added a commit that referenced this issue Dec 10, 2018

hestiacp added a commit to hestiacp/hestiacp that referenced this issue Dec 10, 2018

dpeca added a commit that referenced this issue Dec 27, 2018

Bringing back Better handling of calm-down routine while LE renewing
Accidentally overwritten in f8b4d42 commit
Original commit: 3d8b6a8

Calming down is because #1193 issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment