Skip to content

Zerg dance: writefifo race condition #1486

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pennersr opened this issue Mar 15, 2017 · 3 comments
Closed

Zerg dance: writefifo race condition #1486

pennersr opened this issue Mar 15, 2017 · 3 comments

Comments

@pennersr
Copy link

I have tried to setup a Zerg dance by following the "Art of Graceful Reloading", and could not get it working reliably out of the box when the number of processes is >1.

The problem is that the documentation instructs you to send "1" to the "new.fifo", but when the master is still booting up the workers the "new.fifo" may not be present yet. Example output:

...
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 8699)
spawned uWSGI worker 1 (pid: 8704, cores: 1)
spawned uWSGI worker 2 (pid: 8705, cores: 1)
spawned uWSGI worker 3 (pid: 8706, cores: 1)
spawned uWSGI worker 4 (pid: 8707, cores: 1)
running "writefifo:/var/run/uwsgi/app/foobar/sleeping.fifo Q" (accepting1-once)...
open("/var/run/uwsgi/app/foobar/sleeping.fifo"): No such device or address [core/hooks.c line 134]
running "writefifo:/var/run/uwsgi/app/foobar/running.fifo 2p" (accepting1-once)...
open("/var/run/uwsgi/app/foobar/running.fifo"): No such device or address [core/hooks.c line 134]
running "writefifo:/var/run/uwsgi/app/foobar/new.fifo 1" (accepting1-once)...
open("/var/run/uwsgi/app/foobar/new.fifo"): No such device or address [core/hooks.c line 134]
spawned uWSGI worker 5 (pid: 8708, cores: 1)

A proper run would like like this (here, the workers have all been booted an the write to new.fifo succeeds):

...
spawned uWSGI worker 4 (pid: 8695, cores: 1)
spawned uWSGI worker 5 (pid: 8696, cores: 1)
open("/var/run/uwsgi/app/foobar/sleeping.fifo"): No such device or address [core/hooks.c line 134]
running "writefifo:/var/run/uwsgi/app/foobar/running.fifo 2p" (accepting1-once)...
open("/var/run/uwsgi/app/foobar/running.fifo"): No such device or address [core/hooks.c line 134]
running "writefifo:/var/run/uwsgi/app/foobar/new.fifo 1" (accepting1-once)...
Wed Mar 15 12:53:43 2017 - active master fifo is now /var/run/uwsgi/app/foobar/running.fifo

Apparently, more people have been running into this:

http://lists.unbit.it/pipermail/uwsgi/2014-October/007683.html

The workaround based on the spinningfifo works for me:

https://gist.github.com/unbit/2674313f070673a720e3

Here...

http://lists.unbit.it/pipermail/uwsgi/2014-October/007684.html

... it was stated that:

"Yes for sure, eventually i will merge the hook in the core"

It would be nice if this were tackled -- it would have saved me a lot of time.

@pennersr
Copy link
Author

FWIW -- I have wrapped the gist in a repo so that you can do this:

uwsgi --build-plugin https://github.com/pennersr/uwsgi-spinningfifo

Hope it helps someone ...

@unbit
Copy link
Owner

unbit commented Mar 16, 2017

The spinningfifo action has been added to the core.

@xrmx safe for backport

@wsYdd
Copy link

wsYdd commented Jan 1, 2018

Modify writefifo to spinningfifo solved this problem. Thanks @pennersr

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants