Skip to content

Commit

Permalink
Stop builders gracefully on SIGTERM
Browse files Browse the repository at this point in the history
When SIGTERM is received by the process that is building a build, we only log a
message instead of passing it to the child processes (`pip` running inside the
Docker container, for example).

Otherwise, the `pip` process also receives the SIGTERM and it kills itself,
producing a `exit_status != 0`, registering this as a BuildCommand and making
the whole build to fail. After that, celery "stops gracefully" with a failed build.

This is useful combined with supervisor to make our builders to stop gracefully
in this scenario:

1. supervisorctl stop build
2. celery receives the SIGTERM and logs the warning message
3. supervisor waits for `stopwaitsecs` before sending SIGKILL
  • Loading branch information
humitos committed Apr 24, 2020
1 parent fb01c6d commit 72db6f4
Showing 1 changed file with 9 additions and 0 deletions.
9 changes: 9 additions & 0 deletions readthedocs/projects/tasks.py
Expand Up @@ -404,6 +404,15 @@ def run(self, version_pk): # pylint: disable=arguments-differ
default_retry_delay=7 * 60,
)
def update_docs_task(self, version_pk, *args, **kwargs):
import signal

def sigterm_received(*args, **kwargs):
log.warning('SIGTERM received. Waiting for build to stop gracefully after it finishes.')

# Do not send the SIGTERM signal to childs (pip is automatically killed when
# receives SIGTERM and make the build to fail one command and stop build)
signal.signal(signal.SIGTERM, sigterm_received)

try:
step = UpdateDocsTaskStep(task=self)
return step.run(version_pk, *args, **kwargs)
Expand Down

0 comments on commit 72db6f4

Please sign in to comment.