Reaping process on and on?

[onekiloparsec]

I've struggled with this the whole day. I have two apps made with Django, same circus configuration. One is going fine, the other doesn't and I have no clue.

When I run the command circusd --log-level debug /home/git/conf/circus.ini, I get

2016-10-08 18:00:30 circus[2125] [INFO] Starting master on pid 2125
2016-10-08 18:00:30 circus[2125] [DEBUG] Initializing watchers
2016-10-08 18:00:30 circus[2125] [DEBUG] cmd: gunicorn
2016-10-08 18:00:30 circus[2125] [DEBUG] args: -w 3 -t 60 --pythonpath=. -b 127.0.0.1:8001 app.wsgi
2016-10-08 18:00:30 circus[2125] [DEBUG] process args: ['gunicorn', '-w', '3', '-t', '60', '--    pythonpath=.', '-b', '127.0.0.1:8001', 'app.wsgi']
2016-10-08 18:00:30 circus[2125] [DEBUG] running avatars process [pid 2132]
2016-10-08 18:00:30 circus[2125] [INFO] Arbiter now waiting for commands
2016-10-08 18:00:30 circus[2125] [INFO] avatars started
2016-10-08 18:00:30 circus[2125] [DEBUG] cmd: /usr/bin/python -c 'from circus import stats;     stats.main()' --endpoint tcp://127.0.0.1:5555 --pubsub tcp://127.0.0.1:5556 --statspoint tcp://127.0.0.1:5557
2016-10-08 18:00:30 circus[2125] [DEBUG] args: None
2016-10-08 18:00:30 circus[2125] [DEBUG] process args: ['/usr/bin/python', '-c', 'from circus import stats; stats.main()', '--endpoint', 'tcp://127.0.0.1:5555', '--pubsub', 'tcp://127.0.0.1:5556', '--statspoint', 'tcp://127.0.0.1:5557']
2016-10-08 18:00:30 circus[2125] [DEBUG] running circusd-stats process [pid 2133]
2016-10-08 18:00:30 circus[2125] [INFO] circusd-stats started
2016-10-08 18:00:30 circus[2133] [INFO] Starting the stats streamer
2016-10-08 18:00:30 circus[2125] [DEBUG] got message {"command":"list","properties":{},"id":"6bcfecd02cad460aa0c7b16d169a2406"}
2016-10-08 18:00:30 circus[2125] [DEBUG] got message {"command":"list","properties":{"name":"avatars"},"id":"2790b31cce7942aeb6376bbf867c4b53"}
2016-10-08 18:00:30 circus[2125] [DEBUG] here is the status of the processes []
2016-10-08 18:00:30 circus[2125] [DEBUG] got message {"command":"list","properties":{},"id":"5cf16d4114024f648359d33b167b341f"}
2016-10-08 18:00:30 circus[2125] [DEBUG] got message {"command":"dstats","properties":{},"id":"bb34b81b3b904017b16488c97f4abfc5"}
2016-10-08 18:00:30 circus[2125] [DEBUG] got message {"command":"listsockets","properties":{},"id":"98ee9c4a83e945788d7a9bcce3331083"}
2016-10-08 18:00:35 circus[2125] [DEBUG] reaping process 2132 [avatars]
2016-10-08 18:00:35 circus[2125] [DEBUG] cmd: gunicorn
2016-10-08 18:00:35 circus[2125] [DEBUG] args: -w 3 -t 60 --pythonpath=. -b 127.0.0.1:8001 app.wsgi
2016-10-08 18:00:35 circus[2125] [DEBUG] process args: ['gunicorn', '-w', '3', '-t', '60', '--pythonpath=.', '-b', '127.0.0.1:8001', 'app.wsgi']
2016-10-08 18:00:35 circus[2125] [DEBUG] running avatars process [pid 2138]
2016-10-08 18:00:40 circus[2125] [DEBUG] reaping process 2138 [avatars]
2016-10-08 18:00:40 circus[2125] [DEBUG] cmd: gunicorn
2016-10-08 18:00:40 circus[2125] [DEBUG] args: -w 3 -t 60 --pythonpath=. -b 127.0.0.1:8001 app.wsgi
2016-10-08 18:00:40 circus[2125] [DEBUG] process args: ['gunicorn', '-w', '3', '-t', '60', '--pythonpath=.', '-b', '127.0.0.1:8001', 'app.wsgi']
2016-10-08 18:00:40 circus[2125] [DEBUG] running avatars process [pid 2140]

and it says reaping process on and on with a new PID every time. When I run ps -aux | grep gunicorn every gunicorn process appears to be defunct. But when I start gunicorn manually, everything's fine!

My conf:

[circus]
check_delay = 5
endpoint = tcp://127.0.0.1:5555
pubsub_endpoint = tcp://127.0.0.1:5556
statsd = true

[watcher:avatars]
working_dir = /var/www/html
cmd = gunicorn
args = -w 3 -t 60 --pythonpath=. -b 127.0.0.1:8001 app.wsgi
uid = git
numprocesses = 1
autostart = true
send_hup = true
stdout_stream.class = FileStream
stdout_stream.filename = /home/git/logs/gunicorn.stdout.log
stdout_stream.max_bytes = 10485760
stdout_stream.backup_count = 4
stderr_stream.class = FileStream
stderr_stream.filename = /home/git/logs/gunicorn.stderr.log
stderr_stream.max_bytes = 10485760
stderr_stream.backup_count = 4

[env:avatars]
PATH = /home/git/.virtualenvs/avatars/bin:$PATH
TERM=rxvt-256color
SHELL=/bin/bash
USER=git
LANG=en_US.UTF-8
HOME=/home/git
PYTHONPATH=/home/git/.virtualenvs/avatars/lib/python3.4/site-packages

Any idea? Help or hint would be greatly appreciated! Thanks in advance.

[k4nar]

Do you have any logs of the gunicorn process? Maybe it has a good reason to stop :) .

[onekiloparsec]

Thanks for check, and indeed, I should have done this. Here is the logs:

Traceback (most recent call last):
  File "/home/git/.virtualenvs/avatars/bin/gunicorn", line 7, in <module>
    from gunicorn.app.wsgiapp import run
  File "/home/git/.virtualenvs/avatars/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py", line 10, in <module>
    from gunicorn.app.base import Application
  File "/home/git/.virtualenvs/avatars/lib/python3.4/site-packages/gunicorn/app/base.py", line 9, in <module>
    import traceback
ImportError: No module named 'traceback'

Which is kind of weird, as traceback is a built-in module. ???

[k4nar]

Could be an issue with your PYTHONPATH.

[k4nar]

Hum yeah, PYTHONPATH=/home/git/.virtualenvs/avatars/lib/python3.4/site-packages doesn't look like a valid PYTHONPATH.

[onekiloparsec]

I would be happy to agree with you as your remark makes sense, but it doesn't explain why it works exactly like this on my other server... I'll investigate though.

[onekiloparsec]

By deleting the PYTHONPATH line in my conf.ini file, I managed to get circus working. Thanks for the discussion.

[mangeshlad1986]

It gets fixed by deleting PYTHONPATH but what is the reason behind this? My other deployments are working fine with PYTHONPATH. Is it related to new circus version?