Python VM deadlock

[pcdinh]

BPO	11158
Nosy	@vsajip

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/vsajip'
closed_at = <Date 2011-02-21.22:44:10.552>
created_at = <Date 2011-02-09.08:25:48.722>
labels = ['interpreter-core', 'type-bug', 'invalid']
title = 'Python VM deadlock'
updated_at = <Date 2011-02-21.22:44:10.550>
user = 'https://bugs.python.org/pcdinh'

bugs.python.org fields:

activity = <Date 2011-02-21.22:44:10.550>
actor = 'vinay.sajip'
assignee = 'vinay.sajip'
closed = True
closed_date = <Date 2011-02-21.22:44:10.552>
closer = 'vinay.sajip'
components = ['Interpreter Core']
creation = <Date 2011-02-09.08:25:48.722>
creator = 'pcdinh'
dependencies = []
files = []
hgrepos = []
issue_num = 11158
keywords = []
message_count = 9.0
messages = ['128206', '128232', '128262', '128263', '128265', '128268', '128269', '128286', '129008']
nosy_count = 3.0
nosy_names = ['vinay.sajip', 'marcin.bachry', 'pcdinh']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue11158'
versions = ['Python 2.6']

[pcdinh]

Hi,

I was implementing a multiple process compatible logging which child processes are expected to send all messages to the parent process via a queue and then the parent process serializes and flushes them into a log file. The idea is built around multiprocessing.Queue()

The problem is the first child process is always blocked while subsequent ones can run smoothly. It seems to be blocked on semaphore_wait_signal_trap

Environment
===========
Mac OS X 10.6.5
Python 2.6.6

Reproduction script:
====================
https://gist.github.com/818114

Command to run
====================

python log.py

Console message
===============
Dinh-Phams-MacBook-Pro:task dinhpham$ python log.py
Parent PID 90401
90403
Parent PID 90401
Parent PID 90401
90404
Parent PID 90401
90405
No child process exits
90406
Child process ID 90406
Child process ID 90404
Child process ID 90405
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
Child process PID 90404 will exits now
Child process PID 90406 will exits now
Child process PID 90405 will exits now
The child process 90406 exits
The child process 90405 exits
The child process 90404 exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
^C[90403]
Parent process exits
Dinh-Phams-MacBook-Pro:task dinhpham$

=========> All the process exits except the first child process one (90403)

[vsajip]

I don't believe this is an interpreter or stdlib problem. See this post from September last year:

http://plumberjack.blogspot.com/2010/09/using-logging-with-multiprocessing.html

which shows how to use multiprocessing and logging together with rotating logs. The post includes the code for a complete working script which runs fine with Python 2.6 on Windows and Linux, but which I can't test on Mac OS X/Python 2.6 as I have no access to a Snow Leopard system.

You're welcome to run that script on your system and update this issue with your results - so I'm marking as "invalid" for now (as I don't believe there's a valid Python issue) and "pending" awaiting your feedback. If I don't hear from you in the next week or so, I'll close this issue.

[pcdinh]

I just simplified my script into a smaller one that can be easier to understand and find out what happens

https://gist.github.com/819979

Thanks

[pcdinh]

Sorry, just ignore my latest comment. It is for another issue

[pcdinh]

Hi Vinay,

Your example use a child process to receive log messages. I am implementing a thread-based solution. Do you think that multiprocessing.Queue() has major issues with threads or this problem happens only on Mac OSX implementations?

I still reproduce it quite easily on my system: Mac OS 10.6 and Python 2.6.6

[vsajip]

I don't believe that the problem is about threads vs. processes. In the following gist:

https://gist.github.com/820132

I modified the script so that the main process uses a listener thread, rather than a listener process, to collect the logs. (Basically, I changed multiprocessing.Process to threading.Thread, changed the name of the function to listener_thread from listener_process, added an import for threading and updated the comments. No other changes were made.)

The script in the above gist runs fine on Linux Mint Debian and Windows 7 (Python 2.6 in both cases). Please run it on Snow Leopard and report your findings.

[marcinbachry]

I think it may be related to this bug: http://bugs.python.org/issue6721

[vsajip]

bpo-6721 may well be valid - if this is a symptom of the same issue, then this issue should be closed. I'd like to know if the gist script I referred to works (or not) on Pham Cong Dinh's Snow Leopard machine.