kubernetes keeps complaining one pod is unhealthy, and a ps -ef command shows there is only one python processes left(it is expected to be 2).
The followingf py-spy thread dump shows that the lonely process is dramatic master, and it is waiting for the log watcher thread, which makes sense.
But the problem is, the log watcher thread seems will keep waiting for bytes on the pipe forever, and the master process will not exit, so k8s pod will never get a chance to get restarted.
Process 1: /usr/bin/python /usr/local/bin/dramatiq --queues job --processes 1 --threads 8 robot_processor.app:broker robot_processor.app
Python v3.11.5 (/usr/bin/python3.11)
Thread 1 (idle): "MainThread"
_wait_for_tstate_lock (threading.py:1132)
join (threading.py:1112)
main (dramatiq/cli.py:584)
<module> (dramatiq:8)
Thread 80 (idle): "Thread-1 (watch_logs)"
_recv (multiprocessing/connection.py:382)
_recv_bytes (multiprocessing/connection.py:424)
recv_bytes (multiprocessing/connection.py:219)
watch_logs (dramatiq/cli.py:309)
run (threading.py:975)
_bootstrap_inner (threading.py:1038)
_bootstrap (threading.py:995)
kubernetes keeps complaining one pod is unhealthy, and a
ps -efcommand shows there is only one python processes left(it is expected to be 2).The followingf py-spy thread dump shows that the lonely process is dramatic master, and it is waiting for the log watcher thread, which makes sense.
But the problem is, the log watcher thread seems will keep waiting for bytes on the pipe forever, and the master process will not exit, so k8s pod will never get a chance to get restarted.