Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

--master, atexit, and threads #1609

Open
hynek opened this issue Aug 14, 2017 · 13 comments
Open

--master, atexit, and threads #1609

hynek opened this issue Aug 14, 2017 · 13 comments

Comments

@hynek
Copy link
Contributor

hynek commented Aug 14, 2017 •
edited

Something odd happens if you try to join a thread in an atexit handler.

Consider the following code:

import atexit
import threading


def make_app():
    def application(env, start_response):
        start_response('200 OK', [('Content-Type', 'text/html')])
        return [b"Hello World"]

    e = threading.Event()

    def bg_thread():
        print("thread started")
        e.wait()
        print("thread exiting")

    t = threading.Thread(target=bg_thread, daemon=True)
    t.start()

    @atexit.register
    def cleanup():
        print("start clean up")
        e.set()
        print("msg sent")
        t.join()
        print("end clean up")

    return application

All it does is starting a thread in the background that waits for an event to be set.

If you run it with uwsgi --enable-threads --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()" everything works as espected:

$ uwsgi --enable-threads --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()"
*** Starting uWSGI 2.0.15 (64bit) on [Mon Aug 14 13:50:52 2017] ***
compiled with version: 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42) on 21 July 2017 12:40:07
os: Darwin-16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64
nodename: alpha.local
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 8
current working directory: /Users/hynek/Projects/ssce
detected binary path: /Users/hynek/.virtualenvs/ssce/bin/uwsgi
*** WARNING: you are running uWSGI without its master process manager ***
your processes number limit is 709
your memory page size is 4096 bytes
detected max file descriptor number: 7168
lock engine: OSX spinlocks
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3
Python version: 3.6.1 (default, May  4 2017, 15:25:00)  [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)]
Python main interpreter initialized at 0x7fda0c000000
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 72752 bytes (71 KB) for 1 cores
*** Operational MODE: single process ***
thread started
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x7fda0c000000 pid: 93411 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (and the only) (pid: 93411, cores: 1)
^Cstart clean up
msg sent
thread exiting
end clean up

However if you run it with a master process, it hangs while waiting for the thread which never does receive it:

$ uwsgi --master --enable-threads --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()"
*** Starting uWSGI 2.0.15 (64bit) on [Mon Aug 14 13:53:08 2017] ***
compiled with version: 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42) on 21 July 2017 12:40:07
os: Darwin-16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64
nodename: alpha.local
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 8
current working directory: /Users/hynek/Projects/ssce
detected binary path: /Users/hynek/.virtualenvs/ssce/bin/uwsgi
your processes number limit is 709
your memory page size is 4096 bytes
detected max file descriptor number: 7168
lock engine: OSX spinlocks
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3
Python version: 3.6.1 (default, May  4 2017, 15:25:00)  [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)]
Python main interpreter initialized at 0x7ff63702a000
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145504 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***
thread started
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x7ff63702a000 pid: 93425 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 93425)
spawned uWSGI worker 1 (pid: 93426, cores: 1)
^CSIGINT/SIGQUIT received...killing workers...
start clean up
msg sent
Mon Aug 14 13:54:12 2017 - worker 1 (pid: 93426) is taking too much time to die...NO MERCY !!!
worker 1 buried after 1 seconds
goodbye to uWSGI.

I guess threading is disabled too soon or something? This is relevant to me because I need to clean up my background thread in prometheus_async.
@unbit
Copy link
Owner

unbit commented Aug 14, 2017 •
edited

Hi, this is the expected behaviour with threads + fork (and it is one of the main reasons why golang does not support it ;) Basically the thread is spawned in the master, but the wait happens in the worker for a thread that is not 'runnable'. --lazy-apps should fix your problem. If you need copy on write, spawn the thread in a post_fork hook

@hynek
Copy link
Contributor Author

hynek commented Aug 14, 2017

It does, thanks for the super quick answer! I wish I hadn’t spent so much time on the SSCE. 🙈

@hynek hynek closed this as completed Aug 14, 2017
hynek added a commit to hynek/prometheus-async that referenced this issue Aug 14, 2017
@hynek
Explain threading interaction with uWSGI
804059f 
See also unbit/uwsgi#1609
@hynek
Copy link
Contributor Author

hynek commented Aug 14, 2017

I feel like I need to add for posterity, that for the atexits actually to run on SIGTERM & SIGINT, you need to set --hook-master-start "unix_signal:15 gracefully_kill_them_all" --hook-master-start "unix_signal:2 gracefully_kill_them_all"

(master FIFO is not a good option in containers)

@hynek
Copy link
Contributor Author

hynek commented Aug 14, 2017

Um so I have another question:

Why on earth would uWSGI behave differently when I kill it with Ctrl-C and when I kill it with kill -2 <master pid>?

kill -2 <master pid>:

$ uwsgi --master --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()" --lazy-apps --hook-master-start "unix_signal:2 gracefully_kill_them_all" --enable-threads
*** Starting uWSGI 2.0.15 (64bit) on [Mon Aug 14 16:21:00 2017] ***
compiled with version: 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42) on 21 July 2017 12:40:07
os: Darwin-16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64
nodename: alpha.local
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 8
current working directory: /Users/hynek/Projects/ssce
detected binary path: /Users/hynek/.virtualenvs/ssce/bin/uwsgi
your processes number limit is 709
your memory page size is 4096 bytes
detected max file descriptor number: 7168
lock engine: OSX spinlocks
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3
Python version: 3.6.1 (default, May  4 2017, 15:25:00)  [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)]
Python main interpreter initialized at 0x7fa223002c00
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145504 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 12800)
spawned uWSGI worker 1 (pid: 12801, cores: 1)
running "unix_signal:2 gracefully_kill_them_all" (master-start)...
thread started
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x7fa223002c00 pid: 12801 (default app)
Mon Aug 14 16:21:12 2017 - graceful shutdown triggered...
Gracefully killing worker 1 (pid: 12801)...
start clean up
msg sent
thread exiting
end clean up
worker 1 buried after 1 seconds
goodbye to uWSGI.

Ctrl-C:

$ uwsgi --master --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()" --lazy-apps --hook-master-start "unix_signal:2 gracefully_kill_them_all" --enable-threads
*** Starting uWSGI 2.0.15 (64bit) on [Mon Aug 14 16:21:33 2017] ***
compiled with version: 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42) on 21 July 2017 12:40:07
os: Darwin-16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64
nodename: alpha.local
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 8
current working directory: /Users/hynek/Projects/ssce
detected binary path: /Users/hynek/.virtualenvs/ssce/bin/uwsgi
your processes number limit is 709
your memory page size is 4096 bytes
detected max file descriptor number: 7168
lock engine: OSX spinlocks
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3
Python version: 3.6.1 (default, May  4 2017, 15:25:00)  [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)]
Python main interpreter initialized at 0x7f992e002c00
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145504 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 12855)
spawned uWSGI worker 1 (pid: 12856, cores: 1)
running "unix_signal:2 gracefully_kill_them_all" (master-start)...
thread started
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x7f992e002c00 pid: 12856 (default app)
^CMon Aug 14 16:21:35 2017 - graceful shutdown triggered...
Gracefully killing worker 1 (pid: 12856)...
worker 1 buried after 1 seconds
goodbye to uWSGI.

How does it even know that the signal is coming from my keyboard?

And then if I run it with --threads 2 it gets even more bizarre: kill -2 will trigger the graceful shutdown, however it hangs…until I press Ctrl-C:

$ uwsgi --master --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()" --lazy-apps --hook-master-start "unix_signal:2 gracefully_kill_them_all" --threads 2
*** Starting uWSGI 2.0.15 (64bit) on [Mon Aug 14 16:19:40 2017] ***
compiled with version: 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42) on 21 July 2017 12:40:07
os: Darwin-16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64
nodename: alpha.local
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 8
current working directory: /Users/hynek/Projects/ssce
detected binary path: /Users/hynek/.virtualenvs/ssce/bin/uwsgi
your processes number limit is 709
your memory page size is 4096 bytes
detected max file descriptor number: 7168
lock engine: OSX spinlocks
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3
Python version: 3.6.1 (default, May  4 2017, 15:25:00)  [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)]
Python main interpreter initialized at 0x7f8ce9827400
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 166080 bytes (162 KB) for 2 cores
*** Operational MODE: threaded ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 12739)
spawned uWSGI worker 1 (pid: 12740, cores: 2)
running "unix_signal:2 gracefully_kill_them_all" (master-start)...
thread started
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x7f8ce9827400 pid: 12740 (default app)
Mon Aug 14 16:19:52 2017 - graceful shutdown triggered...
Gracefully killing worker 1 (pid: 12740)...
^Cstart clean up
msg sent
thread exiting
end clean up
worker 1 buried after 29 seconds
goodbye to uWSGI.

Could you shed some light here what is going on? Is it even possible to run WSGI application with a side-car thread?

@hynek hynek reopened this Aug 14, 2017
@unbit
Copy link
Owner

unbit commented Aug 14, 2017

Can you try on a Linux system ? I know it may look foolish but it is not the first time signals + pthreads result in weird behaviours on osx :(

@unbit
Copy link
Owner

unbit commented Aug 15, 2017

In the mean time i can confirm that pthread cancelation does not work as expected on OSX. So you should add the option --no-threads-wait if you plan to spawn multiple threads on darwin. Basically with this option the worker is killed without waiting for threads cooperation.

@hynek
Copy link
Contributor Author

hynek commented Aug 15, 2017 •
edited

OK Linux time! It’s an Ubuntu Trusty VM running in VMWare.

--enable-threads + kill -2

$ uwsgi --master --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()" \
--lazy-apps --hook-master-start "unix_signal:2 gracefully_kill_them_all" \
--enable-threads
*** Starting uWSGI 2.0.15 (64bit) on [Tue Aug 15 07:14:32 2017] ***
compiled with version: 4.8.4 on 15 August 2017 07:11:01
os: Linux-3.13.0-110-generic #157-Ubuntu SMP Mon Feb 20 11:54:05 UTC 2017
nodename: trusty64
machine: x86_64
clock source: unix
detected number of CPU cores: 2
current working directory: /home/vagrant/Projects/ssce
detected binary path: /home/vagrant/.virtualenvs/ssce/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your processes number limit is 15832
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3
Python version: 3.6.2 (default, Jul 29 2017, 00:00:00)  [GCC 4.8.4]
Python main interpreter initialized at 0x11a2820
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145536 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 12720)
spawned uWSGI worker 1 (pid: 12721, cores: 1)
running "unix_signal:2 gracefully_kill_them_all" (master-start)...
thread started
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x11a2820 pid: 12721 (default app)
Tue Aug 15 07:14:43 2017 - graceful shutdown triggered...
Gracefully killing worker 1 (pid: 12721)...
start clean up
msg sent
thread exiting
end clean up
worker 1 buried after 1 seconds
goodbye to uWSGI.

Works!

--enable-threads + Ctrl-C

$ uwsgi --master --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()" \
--lazy-apps --hook-master-start "unix_signal:2 gracefully_kill_them_all" \
--enable-threads
*** Starting uWSGI 2.0.15 (64bit) on [Tue Aug 15 07:14:05 2017] ***
compiled with version: 4.8.4 on 15 August 2017 07:11:01
os: Linux-3.13.0-110-generic #157-Ubuntu SMP Mon Feb 20 11:54:05 UTC 2017
nodename: trusty64
machine: x86_64
clock source: unix
detected number of CPU cores: 2
current working directory: /home/vagrant/Projects/ssce
detected binary path: /home/vagrant/.virtualenvs/ssce/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your processes number limit is 15832
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3
Python version: 3.6.2 (default, Jul 29 2017, 00:00:00)  [GCC 4.8.4]
Python main interpreter initialized at 0xc55820
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145536 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 12712)
spawned uWSGI worker 1 (pid: 12713, cores: 1)
running "unix_signal:2 gracefully_kill_them_all" (master-start)...
thread started
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0xc55820 pid: 12713 (default app)
^CTue Aug 15 07:14:09 2017 - graceful shutdown triggered...
start clean up
msg sent
thread exiting
end clean up
worker 1 buried after 1 seconds
goodbye to uWSGI.

Works!

--threads 2 + kill -2

uwsgi --master --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()" --lazy-apps --hook-master-start "unix_signal:2 gracefully_kill_them_all" --threads 2
*** Starting uWSGI 2.0.15 (64bit) on [Tue Aug 15 07:21:59 2017] ***
compiled with version: 4.8.4 on 15 August 2017 07:11:01
os: Linux-3.13.0-110-generic #157-Ubuntu SMP Mon Feb 20 11:54:05 UTC 2017
nodename: trusty64
machine: x86_64
clock source: unix
detected number of CPU cores: 2
current working directory: /home/vagrant/Projects/ssce
detected binary path: /home/vagrant/.virtualenvs/ssce/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your processes number limit is 15832
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3
Python version: 3.6.2 (default, Jul 29 2017, 00:00:00)  [GCC 4.8.4]
Python main interpreter initialized at 0x1622840
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 166144 bytes (162 KB) for 2 cores
*** Operational MODE: threaded ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 12804)
spawned uWSGI worker 1 (pid: 12805, cores: 2)
running "unix_signal:2 gracefully_kill_them_all" (master-start)...
thread started
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x1622840 pid: 12805 (default app)
Tue Aug 15 07:22:04 2017 - graceful shutdown triggered...
Gracefully killing worker 1 (pid: 12805)...
start clean up
msg sent
thread exiting
end clean up
worker 1 buried after 1 seconds
goodbye to uWSGI.

Seems to work but I think I had it fail occasionally. Not 100% sure tho.

--threads 2 + Ctrl-C

$ uwsgi --master --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()" \
--lazy-apps --hook-master-start "unix_signal:2 gracefully_kill_them_all" \
 --threads 2
*** Starting uWSGI 2.0.15 (64bit) on [Tue Aug 15 07:12:06 2017] ***
compiled with version: 4.8.4 on 15 August 2017 07:11:01
os: Linux-3.13.0-110-generic #157-Ubuntu SMP Mon Feb 20 11:54:05 UTC 2017
nodename: trusty64
machine: x86_64
clock source: unix
detected number of CPU cores: 2
current working directory: /home/vagrant/Projects/ssce
detected binary path: /home/vagrant/.virtualenvs/ssce/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your processes number limit is 15832
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3
Python version: 3.6.2 (default, Jul 29 2017, 00:00:00)  [GCC 4.8.4]
Python main interpreter initialized at 0x1e94840
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 166144 bytes (162 KB) for 2 cores
*** Operational MODE: threaded ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 12705)
spawned uWSGI worker 1 (pid: 12706, cores: 2)
running "unix_signal:2 gracefully_kill_them_all" (master-start)...
thread started
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x1e94840 pid: 12706 (default app)
^CTue Aug 15 07:12:08 2017 - graceful shutdown triggered...
start clean up
msg sent
^Cworker 1 buried after 27 seconds
goodbye to uWSGI.

So it apparently starts the cleanup but the join fails because it needs a second Ctrl-C.

To me, this looks awfully like some kind of race condition (who’d thought with threads :D)? Especially because very rarely it works too.

I’ve also tried sprinkle in a bunch of print(threading.enumerate()) that indicate that my thread isn't running anymore and instead there's [<_MainThread(b'uWSGIWorker1Core0', stopped 140602576090944)>, <_DummyThread(b'uWSGIWorker1Core1', started daemon 140602515154688)>]. _DummyThread!?

I don’t know if that helps you at all…here's the SSCE again if you wanna run it yourself.

from __future__ import print_function

import atexit
import threading


def make_app():
    def application(env, start_response):
        start_response('200 OK', [('Content-Type', 'text/html')])
        return [b"Hello World"]

    e = threading.Event()

    def bg_thread():
        print("thread started")
        e.wait()
        print("thread exiting")

    print(threading.enumerate())
    t = threading.Thread(target=bg_thread, name="bg-thread")
    
    t.daemon = True
    t.start()
    print(threading.enumerate())

    @atexit.register
    def cleanup():
        print("start clean up")
        print(threading.enumerate())
        e.set()
        print("msg sent")
        print(threading.enumerate())
        print(threading.current_thread(), t)
        t.join()
        print("end clean up")

    return application

Setting deamon to False does also some really funky stuff but let’s focus on one thing. ;)

@unbit
Copy link
Owner

unbit commented Aug 15, 2017

Does using --no-threads-wait improves the situation ? (both in Linux and OSX)

@hynek
Copy link
Contributor Author

hynek commented Aug 15, 2017

macOS

$ uwsgi --master --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()" \
--lazy-apps --hook-master-start "unix_signal:2 gracefully_kill_them_all" \
--threads 2 --no-threads-wait
*** Starting uWSGI 2.0.15 (64bit) on [Tue Aug 15 08:05:20 2017] ***
compiled with version: 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42) on 21 July 2017 12:40:07
os: Darwin-16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64
nodename: alpha.local
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 8
current working directory: /Users/hynek/Projects/ssce
detected binary path: /Users/hynek/.virtualenvs/ssce/bin/uwsgi
your processes number limit is 709
your memory page size is 4096 bytes
detected max file descriptor number: 7168
lock engine: OSX spinlocks
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3
Python version: 3.6.1 (default, May  4 2017, 15:25:00)  [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)]
Python main interpreter initialized at 0x7f80ee017600
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 166080 bytes (162 KB) for 2 cores
*** Operational MODE: threaded ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 31512)
spawned uWSGI worker 1 (pid: 31513, cores: 2)
running "unix_signal:2 gracefully_kill_them_all" (master-start)...
[<_MainThread(b'uWSGIWorker1Core0', started 140736085013440)>]
thread started
[<_MainThread(b'uWSGIWorker1Core0', started 140736085013440)>, <Thread(bg-thread, started daemon 123145463066624)>]
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x7f80ee017600 pid: 31513 (default app)
^CTue Aug 15 08:05:21 2017 - graceful shutdown triggered...
Gracefully killing worker 1 (pid: 31513)...
worker 1 buried after 1 seconds
goodbye to uWSGI.

Shutdown works but atexit is not executed.

Linux

uwsgi --master --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()" \
--lazy-apps --hook-master-start "unix_signal:2 gracefully_kill_them_all" \
--threads 2 --no-threads-wait
*** Starting uWSGI 2.0.15 (64bit) on [Tue Aug 15 08:07:07 2017] ***
compiled with version: 4.8.4 on 15 August 2017 07:11:01
os: Linux-3.13.0-110-generic #157-Ubuntu SMP Mon Feb 20 11:54:05 UTC 2017
nodename: trusty64
machine: x86_64
clock source: unix
detected number of CPU cores: 2
current working directory: /home/vagrant/Projects/ssce
detected binary path: /home/vagrant/.virtualenvs/ssce/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your processes number limit is 15832
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3
Python version: 3.6.2 (default, Jul 29 2017, 00:00:00)  [GCC 4.8.4]
Python main interpreter initialized at 0x1493890
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 166144 bytes (162 KB) for 2 cores
*** Operational MODE: threaded ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 13787)
spawned uWSGI worker 1 (pid: 13788, cores: 2)
running "unix_signal:2 gracefully_kill_them_all" (master-start)...
[<_MainThread(b'uWSGIWorker1Core0', started 140100408424256)>]
thread started
[<_MainThread(b'uWSGIWorker1Core0', started 140100408424256)>, <Thread(bg-thread, started daemon 140100356142848)>]
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x1493890 pid: 13788 (default app)
^CTue Aug 15 08:07:10 2017 - graceful shutdown triggered...
start clean up
[<_MainThread(b'uWSGIWorker1Core0', stopped 140100408424256)>, <_DummyThread(b'uWSGIWorker1Core1', started daemon 140100347750144)>]
msg sent
[<_MainThread(b'uWSGIWorker1Core0', stopped 140100408424256)>, <_DummyThread(b'uWSGIWorker1Core1', started daemon 140100347750144)>]
Tue Aug 15 08:08:11 2017 - worker 1 (pid: 13788) is taking too much time to die...NO MERCY !!!
worker 1 buried after 1 seconds
goodbye to uWSGI.

Shutdown starts (msg sent) but hangs there, the thread doesn't seem to exist in cleanup anymore – given the prints.

@unbit
Copy link
Owner

unbit commented Aug 15, 2017

Oh, this is interesting, so on the mac ctrl-c has some magic behaviour that needs to be tackled :) but on Linux there is something more strange happening as it looks like the worker is blocked in the t.join() part. Does it happen when hitting ctrl-c, when sending -2 to the master or in both cases ?

@hynek
Copy link
Contributor Author

hynek commented Aug 15, 2017

So all Linux now:

Ctrl-C

$ uwsgi --master --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()" \
--lazy-apps --hook-master-start "unix_signal:2 gracefully_kill_them_all" \
--threads 2 --no-threads-wait
*** Starting uWSGI 2.0.15 (64bit) on [Tue Aug 15 08:25:33 2017] ***
compiled with version: 4.8.4 on 15 August 2017 07:11:01
os: Linux-3.13.0-110-generic #157-Ubuntu SMP Mon Feb 20 11:54:05 UTC 2017
nodename: trusty64
machine: x86_64
clock source: unix
detected number of CPU cores: 2
current working directory: /home/vagrant/Projects/ssce
detected binary path: /home/vagrant/.virtualenvs/ssce/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your processes number limit is 15832
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3
Python version: 3.6.2 (default, Jul 29 2017, 00:00:00)  [GCC 4.8.4]
Python main interpreter initialized at 0x2100890
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 166144 bytes (162 KB) for 2 cores
*** Operational MODE: threaded ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 13863)
spawned uWSGI worker 1 (pid: 13864, cores: 2)
running "unix_signal:2 gracefully_kill_them_all" (master-start)...
[<_MainThread(b'uWSGIWorker1Core0', started 140571148867392)>]
thread started
[<_MainThread(b'uWSGIWorker1Core0', started 140571148867392)>, <Thread(bg-thread, started daemon 140571096585984)>]
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x2100890 pid: 13864 (default app)
^CTue Aug 15 08:25:34 2017 - graceful shutdown triggered...
start clean up
[<_MainThread(b'uWSGIWorker1Core0', stopped 140571148867392)>, <_DummyThread(b'uWSGIWorker1Core1', started daemon 140571088193280)>]
msg sent
[<_MainThread(b'uWSGIWorker1Core0', stopped 140571148867392)>, <_DummyThread(b'uWSGIWorker1Core1', started daemon 140571088193280)>]
^C
worker 1 buried after 4 seconds
goodbye to uWSGI.

→ the thread isn’t there and it hangs

kill -2

uwsgi --master --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()" \
--lazy-apps --hook-master-start "unix_signal:2 gracefully_kill_them_all" \
--threads 2 --no-threads-wait
*** Starting uWSGI 2.0.15 (64bit) on [Tue Aug 15 08:25:39 2017] ***
compiled with version: 4.8.4 on 15 August 2017 07:11:01
os: Linux-3.13.0-110-generic #157-Ubuntu SMP Mon Feb 20 11:54:05 UTC 2017
nodename: trusty64
machine: x86_64
clock source: unix
detected number of CPU cores: 2
current working directory: /home/vagrant/Projects/ssce
detected binary path: /home/vagrant/.virtualenvs/ssce/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your processes number limit is 15832
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3
Python version: 3.6.2 (default, Jul 29 2017, 00:00:00)  [GCC 4.8.4]
Python main interpreter initialized at 0x23a7890
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 166144 bytes (162 KB) for 2 cores
*** Operational MODE: threaded ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 13868)
spawned uWSGI worker 1 (pid: 13869, cores: 2)
running "unix_signal:2 gracefully_kill_them_all" (master-start)...
[<_MainThread(b'uWSGIWorker1Core0', started 139755514599232)>]
thread started
[<_MainThread(b'uWSGIWorker1Core0', started 139755514599232)>, <Thread(bg-thread, started daemon 139755462317824)>]
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x23a7890 pid: 13869 (default app)
Tue Aug 15 08:25:46 2017 - graceful shutdown triggered...
Gracefully killing worker 1 (pid: 13869)...
start clean up
[<Thread(bg-thread, started daemon 139755462317824)>, <_DummyThread(b'uWSGIWorker1Core1', started daemon 139755453925120)>]
msg sent
thread exiting
[<Thread(bg-thread, started daemon 139755462317824)>, <_DummyThread(b'uWSGIWorker1Core1', started daemon 139755453925120)>]
end clean up
worker 1 buried after 1 seconds
goodbye to uWSGI.

→ The thread is there and it works.

I gotta run now, so I suggest you play with my SSCE I’ve posted before, it has no dependencies and works on both Python 2 and Python 3. :)

@hynek
Copy link
Contributor Author

hynek commented Jan 6, 2018

JFTR, I remembered why Ctrl-C and kill -2 work differently: doing a Ctrl-C fires a SIGINT to all foreground processes. That seems to confuse things.

@kevin19930919
Copy link

I recently working on a nginx+uwsgi+flask+golang web service. Before i add some golang shared library for python ,it all work fine.But if i call golang shared library in my python script,it stuck without any error, just no response from golang.(it works fine if i call golang shared library with python interpreter)
after i add lazy-apps and --no-threads-wait , the stucking problem fixed!!! I think it maybe some problem with gorotuine,but i not sure the main reason. Can anyone explain this,or maybe give me some keyword that i can try to find

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@hynek @unbit @kevin19930919