WorkerTmp unlinking causes silent failure #1388

Open
caervs opened this Issue Nov 10, 2016 · 4 comments

Projects

None yet

3 participants

@caervs
caervs commented Nov 10, 2016

I was running into issue #407 which was fixed when I took the advice there and set a different --worker-tmp-dir.

Unfortunately the problem was hard to diagnose because it appears the exception causes the arbiter's logger to stop working so the exception never gets logged. Even with debug logs I just saw

[2016-11-10 01:04:21 +0000] [624] [INFO] Starting gunicorn 19.6.0
[2016-11-10 01:04:21 +0000] [624] [DEBUG] Arbiter booted
[2016-11-10 01:04:21 +0000] [624] [INFO] Listening at: http://0.0.0.0:1000 (624)
[2016-11-10 01:04:21 +0000] [624] [INFO] Using worker: sync
[2016-11-10 01:04:21 +0000] [629] [INFO] Booting worker with pid: 629
[2016-11-10 01:04:21 +0000] [630] [INFO] Booting worker with pid: 630
[2016-11-10 01:04:21 +0000] [631] [INFO] Booting worker with pid: 631
[2016-11-10 01:04:21 +0000] [634] [INFO] Booting worker with pid: 634
[2016-11-10 01:04:21 +0000] [624] [DEBUG] 4 workers
[2016-11-10 01:04:23 +0000] [641] [INFO] Booting worker with pid: 641
[2016-11-10 01:04:23 +0000] [624] [DEBUG] 3 workers
[2016-11-10 01:04:23 +0000] [642] [INFO] Booting worker with pid: 642
[2016-11-10 01:04:23 +0000] [643] [INFO] Booting worker with pid: 643

I basically diagnosed by taking the line: https://github.com/benoitc/gunicorn/blob/master/gunicorn/arbiter.py#L557
and wrapping it in a try/except block and printed the traceback.

This is both on gunicorn 19.1.0 and 19.6.0

My environment is Mac OS X running Docker for Mac 1.12.3-rc1 and the issue occurs in a container where /tmp/ has been mounted in from the host.

@benoitc
Owner
benoitc commented Nov 17, 2016

was is the fs layer used by docker?

@tilgovi
Collaborator
tilgovi commented Dec 20, 2016

@caervs what is the error you get?

@caervs
caervs commented Jan 5, 2017

@tilgovi I was seeing the same stacktrace from #407

It looked like

Traceback (most recent call last):
  File "/cygdrive/f/dev/beerulatrix_project/venv/lib/python2.6/site-packages/gunicorn/arbiter.py", line 459, in spawn_worker
    worker.init_process()
  File "/cygdrive/f/dev/beerulatrix_project/venv/lib/python2.6/site-packages/gunicorn/workers/base.py", line 103, in init_process
    self.run()
  File "/cygdrive/f/dev/beerulatrix_project/venv/lib/python2.6/site-packages/gunicorn/workers/sync.py", line 26, in run
    self.notify()
  File "/cygdrive/f/dev/beerulatrix_project/venv/lib/python2.6/site-packages/gunicorn/workers/base.py", line 64, in notify
    self.tmp.notify()
  File "/cygdrive/f/dev/beerulatrix_project/venv/lib/python2.6/site-packages/gunicorn/workers/workertmp.py", line 34, in notify
    os.fchmod(self._tmp.fileno(), self.spinner)
OSError: [Errno 2] No such file or directory

I can see about getting a trace from my local machine

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment