New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Zerg dance: writefifo race condition #1486

Closed
pennersr opened this Issue Mar 15, 2017 · 3 comments

Comments

Projects
None yet
3 participants
@pennersr

pennersr commented Mar 15, 2017

I have tried to setup a Zerg dance by following the "Art of Graceful Reloading", and could not get it working reliably out of the box when the number of processes is >1.

The problem is that the documentation instructs you to send "1" to the "new.fifo", but when the master is still booting up the workers the "new.fifo" may not be present yet. Example output:

...
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 8699)
spawned uWSGI worker 1 (pid: 8704, cores: 1)
spawned uWSGI worker 2 (pid: 8705, cores: 1)
spawned uWSGI worker 3 (pid: 8706, cores: 1)
spawned uWSGI worker 4 (pid: 8707, cores: 1)
running "writefifo:/var/run/uwsgi/app/foobar/sleeping.fifo Q" (accepting1-once)...
open("/var/run/uwsgi/app/foobar/sleeping.fifo"): No such device or address [core/hooks.c line 134]
running "writefifo:/var/run/uwsgi/app/foobar/running.fifo 2p" (accepting1-once)...
open("/var/run/uwsgi/app/foobar/running.fifo"): No such device or address [core/hooks.c line 134]
running "writefifo:/var/run/uwsgi/app/foobar/new.fifo 1" (accepting1-once)...
open("/var/run/uwsgi/app/foobar/new.fifo"): No such device or address [core/hooks.c line 134]
spawned uWSGI worker 5 (pid: 8708, cores: 1)

A proper run would like like this (here, the workers have all been booted an the write to new.fifo succeeds):

...
spawned uWSGI worker 4 (pid: 8695, cores: 1)
spawned uWSGI worker 5 (pid: 8696, cores: 1)
open("/var/run/uwsgi/app/foobar/sleeping.fifo"): No such device or address [core/hooks.c line 134]
running "writefifo:/var/run/uwsgi/app/foobar/running.fifo 2p" (accepting1-once)...
open("/var/run/uwsgi/app/foobar/running.fifo"): No such device or address [core/hooks.c line 134]
running "writefifo:/var/run/uwsgi/app/foobar/new.fifo 1" (accepting1-once)...
Wed Mar 15 12:53:43 2017 - active master fifo is now /var/run/uwsgi/app/foobar/running.fifo

Apparently, more people have been running into this:

http://lists.unbit.it/pipermail/uwsgi/2014-October/007683.html

The workaround based on the spinningfifo works for me:

https://gist.github.com/unbit/2674313f070673a720e3

Here...

http://lists.unbit.it/pipermail/uwsgi/2014-October/007684.html

... it was stated that:

"Yes for sure, eventually i will merge the hook in the core"

It would be nice if this were tackled -- it would have saved me a lot of time.

@pennersr

This comment has been minimized.

Show comment
Hide comment
@pennersr

pennersr Mar 15, 2017

FWIW -- I have wrapped the gist in a repo so that you can do this:

uwsgi --build-plugin https://github.com/pennersr/uwsgi-spinningfifo

Hope it helps someone ...

pennersr commented Mar 15, 2017

FWIW -- I have wrapped the gist in a repo so that you can do this:

uwsgi --build-plugin https://github.com/pennersr/uwsgi-spinningfifo

Hope it helps someone ...

@unbit

This comment has been minimized.

Show comment
Hide comment
@unbit

unbit Mar 16, 2017

Owner

The spinningfifo action has been added to the core.

@xrmx safe for backport

Owner

unbit commented Mar 16, 2017

The spinningfifo action has been added to the core.

@xrmx safe for backport

@wsYdd

This comment has been minimized.

Show comment
Hide comment
@wsYdd

wsYdd Jan 1, 2018

Modify writefifo to spinningfifo solved this problem. Thanks @pennersr

wsYdd commented Jan 1, 2018

Modify writefifo to spinningfifo solved this problem. Thanks @pennersr

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment