New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restarting nicely with supervisord #88

Closed
miohtama opened this Issue Feb 8, 2015 · 14 comments

Comments

Projects
None yet
2 participants
@miohtama
Contributor

miohtama commented Feb 8, 2015

I am trying to make supervisorctl restart to work with huey. At the moment, restart doesn't kill the old process, but still creates another running huey instance.

Below is a terminal capture showing huey is duplicated on restart:

(venv)root@Ubuntu-1204-precise-64-minimal /srv/django/tatianastore # ps -Af|grep -i huey
root     11260     1  0 03:46 ?        00:00:01 python manage.py run_huey --periodic
root     13367  1166  0 03:55 ?        00:00:00 /bin/sh /srv/django/tatianastore/manage_wrapper.sh run_huey --periodic
root     13370 13367  6 03:55 ?        00:00:00 python manage.py run_huey --periodic
root     13378 10409  0 03:55 pts/1    00:00:00 grep --color=auto -i huey

(venv)root@Ubuntu-1204-precise-64-minimal /srv/django/tatianastore # supervisorctl restart tatianastore_huey
tatianastore_huey: stopped
tatianastore_huey: started

(venv)root@Ubuntu-1204-precise-64-minimal /srv/django/tatianastore # ps -Af|grep -i hueyroot     11260     1  0 03:46 ?        00:00:01 python manage.py run_huey --periodic
root     13370     1  2 03:55 ?        00:00:00 python manage.py run_huey --periodic
root     13401  1166  0 03:55 ?        00:00:00 /bin/sh /srv/django/tatianastore/manage_wrapper.sh run_huey --periodic
root     13404 13401 14 03:55 ?        00:00:00 python manage.py run_huey --periodic
root     13433 10409  0 03:55 pts/1    00:00:00 grep --color=auto -i huey

My supervisor config: https://github.com/miohtama/LibertyMusicStore/blob/master/conf/supervisor.conf

Which UNIX signal supervisor needs to send to huey in order to properly stop it? Or are there any special considerations regarding daemonizing huey process?

@coleifer

This comment has been minimized.

Owner

coleifer commented Feb 9, 2015

Hmm, I'm not sure as I haven't run into this particular bug before. Technically huey isn't ever daemonized, it just runs in the foreground and supervisor keeps an eye on it. I checked on my server and when I restarted the PID changed and there was only ever 1 instance running.

[program:blog_queue]
directory=/whatever/path/
command=/path/to/bin/huey_consumer.py blog.main.huey --logfile=/path/to/huey-blog.log -S 300
user=whoever
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/path/to/huey_supervisor.out.log
stderr_logfile=/path/to/huey_supervisor.out.err
environment=PYTHONPATH="/whatever:$PYTHONPATH"
@coleifer

This comment has been minimized.

Owner

coleifer commented Feb 9, 2015

I'm not quite sure what to do here...I'm unable to replicate this bug locally. Closing for now.

@coleifer coleifer closed this Feb 9, 2015

@miohtama

This comment has been minimized.

Contributor

miohtama commented Feb 11, 2015

Thanks! I'll keep investigating why it doesn't restart properly.

Just a question: What (should) happen if there is some periodical task running / hung and the process gets SIGTERM?

@coleifer

This comment has been minimized.

Owner

coleifer commented Feb 11, 2015

Actually that may be the problem... Huey will keep running until the long-running task finishes.

@coleifer coleifer reopened this Feb 11, 2015

@miohtama

This comment has been minimized.

Contributor

miohtama commented Feb 11, 2015

In my case, there should not be any long running tasks or those tasks should terminate eventually. However I have seen huey instances over few days old.

I'll try to dump the huey state using Python 3 faulthandler ( https://docs.python.org/3/library/faulthandler.html )

@coleifer

This comment has been minimized.

Owner

coleifer commented Feb 11, 2015

OK, that's good to know about the tasks in your app. I'll audit the code and see if I can find any issues.

@coleifer

This comment has been minimized.

Owner

coleifer commented Feb 11, 2015

One thing, do you mind sharing the invocation you use to start huey consumer?

@miohtama

This comment has been minimized.

Contributor

miohtama commented Feb 12, 2015

Here is the exact supervisor config I am using (containing the invocation):

https://github.com/miohtama/LibertyMusicStore/blob/master/conf/supervisor.conf

@coleifer

This comment has been minimized.

Owner

coleifer commented Feb 12, 2015

Thanks @miohtama -- I'm going to take a look and see what I can see. When you run huey using the following invocation directly from the shell what happens when you press Ctrl+C?

$ cd /srv/django/tatianastore
$ /srv/django/tatianastore/manage_wrapper.sh run_huey --periodic
# Wait til huey starts up, then press Ctrl+C.  What happens?
@coleifer

This comment has been minimized.

Owner

coleifer commented Feb 12, 2015

Alright I was able to replicate this locally and the issue is the fact that you're using a bash script to run the consumer. When I used a bash script to invoke the consumer, the consumer would continue running indefinitely. When I replaced the bash script with a direct invocation of the consumer, it worked as expected.

So, here is how I would suggest changing your config -- I believe this will accomplish the same thing as your wrapper:

[program:tatianastore_huey]
command=/srv/django/tatianastore/bin/python /srv/django/tatianastore/manage.py run_huey --periodic

@coleifer coleifer closed this Feb 12, 2015

@coleifer

This comment has been minimized.

Owner

coleifer commented Feb 12, 2015

@miohtama

This comment has been minimized.

Contributor

miohtama commented Feb 12, 2015

Ah, of course! I should have known this as I have run across the same issue couple of time during the years.

The rationale creating the (irritating) wrapper script is that venv module shipped with Python 3.4 on Ubuntu 12.04 is broken. It incorrectly fails to set PYTHONPATH and behaves differently than real virtualenv. Thus, one needs to run the script and source the environment.

However the alternative solution is not to use python -m venv or supplied virtualenv, but simply install virtualenv from PyPi and override the system .deb installation.

@miohtama

This comment has been minimized.

Contributor

miohtama commented Feb 12, 2015

Thanks for the help once again!

@coleifer

This comment has been minimized.

Owner

coleifer commented Jul 7, 2017

The info in this comment is a bit outdated: when huey consumer receives SIGTERM, it will not wait for any currently-executing tasks to finish.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment