Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multiprocess RQ workers (using supervisor) #4371

Merged
merged 11 commits into from
Jan 1, 2020
Merged

Multiprocess RQ workers (using supervisor) #4371

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
153bcde
launch and monitor multiple workers using supervisor
Nov 19, 2019
4d1ae28
run supervisord in non-daemon mode
Nov 19, 2019
1fb1a7e
redirect all output to stdout/stderr
Nov 19, 2019
fdbbd13
no need to log supervisord's output because it is redirected to stdou…
Nov 19, 2019
22065a7
updated and less brittle healthcheck
Oct 10, 2019
092eb6c
add supervisor healthchecks
Nov 21, 2019
d0c61b3
Merge branch 'master' into multi-process-rq-workers-using-supervisor
Nov 21, 2019
76909ea
remove redundant supervisor installation as it is installed by pip
Nov 24, 2019
1d946ae
add a 5 minute check gate
Nov 24, 2019
b9fcb4d
Merge branch 'master' into multi-process-rq-workers-using-supervisor
Dec 30, 2019
dd6b3d2
Merge branch 'master' into multi-process-rq-workers-using-supervisor
Jan 1, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 2 additions & 1 deletion Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,8 @@ RUN apt-get update && \
libssl-dev \
default-libmysqlclient-dev \
freetds-dev \
libsasl2-dev && \
libsasl2-dev \
supervisor && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*

Expand Down
12 changes: 10 additions & 2 deletions bin/docker-entrypoint
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,21 @@ dev_scheduler() {
worker() {
echo "Starting RQ worker..."

exec /app/manage.py rq worker $QUEUES
start_worker
}

dev_worker() {
echo "Starting dev RQ worker..."

exec watchmedo auto-restart --directory=./redash/ --pattern=*.py --recursive -- ./manage.py rq worker $QUEUES
start_worker "watchmedo auto-restart --directory=./redash/ --pattern=*.py --recursive -- "
}

start_worker() {
export EXECUTION_PREFIX=$1
export WORKERS_COUNT=${WORKERS_COUNT:-2}
export QUEUES=${QUEUES:-}

supervisord -c worker.conf
}

dev_celery_worker() {
Expand Down
1 change: 0 additions & 1 deletion docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,6 @@ services:
target: /app
depends_on:
- server
tty: true
environment:
PYTHONUNBUFFERED: 0
REDASH_LOG_LEVEL: "INFO"
Expand Down
17 changes: 17 additions & 0 deletions worker.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
[supervisord]
logfile=/tmp/supervisord.log
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to update this to stdout as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fdbbd13

pidfile=/tmp/supervisord.pid
nodaemon=true

[program:worker]
command=%(ENV_EXECUTION_PREFIX)s ./manage.py rq worker %(ENV_QUEUES)s
process_name=%(program_name)s-%(process_num)s
numprocs=%(ENV_WORKERS_COUNT)s
directory=/app
stopsignal=TERM
autostart=true
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0