The multiprocessing.Queue in Python does not function properly on Windows 10.

[rin-0]

Bug report

Bug description:

I am using the following version of Python:
Python 3.11.3 (tags/v3.11.3:f3909b8, Apr 4 2023, 23:49:59) [MSC v.1934 64 bit (AMD64)] on win32.
Here is my code:

import multiprocessing
def sub_process(queue):
    print("Sub process started")
    data = queue.get()
    print(f"Sub process received {data} from main process")
    queue.put("Hello from sub process")
    print("Sub process finished")

if __name__ == "__main__":
    queue = multiprocessing.Queue()
    p = multiprocessing.Process(target=sub_process, args=(queue,))
    p.start()
    queue.put("Hello from main process")
    print("Main process sent data to sub process")
    data = queue.get()
    print(f"Main process received {data} from sub process")
    p.join()

This code works normally in Linux with Python 3.10.4. However, it does not work properly on Windows and gets stuck at data = queue.get() in the sub_process method.

PS D:\project\my-py-project> python .\test.py
Main process sent data to sub process
Main process received Hello from main process from sub process
Sub process started
Process Process-1:
Traceback (most recent call last):
  File "C:\me\software\python\python311\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "C:\me\software\python\python311\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "D:\project\my-py-project\test.py", line 4, in sub_process
    data = queue.get()
           ^^^^^^^^^^^
  File "C:\me\software\python\python311\Lib\multiprocessing\queues.py", line 103, in get
    res = self._recv_bytes()
          ^^^^^^^^^^^^^^^^^^
  File "C:\me\software\python\python311\Lib\multiprocessing\connection.py", line 215, in recv_bytes
    buf = self._recv_bytes(maxlength)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\me\software\python\python311\Lib\multiprocessing\connection.py", line 304, in _recv_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
Traceback (most recent call last):
  File "D:\project\my-py-project\test.py", line 17, in <module>
    p.join()
  File "C:\me\software\python\python311\Lib\multiprocessing\process.py", line 149, in join
    res = self._popen.wait(timeout)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\me\software\python\python311\Lib\multiprocessing\popen_spawn_win32.py", line 109, in wait
    res = _winapi.WaitForSingleObject(int(self._handle), msecs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt

CPython versions tested on:

3.11

Operating systems tested on:

Windows

[gaogaotiantian]

This is just a dead lock, not a bug in Python. Your main process and subproces shared a single queue to communicate, but there's no guarantee when the two processes will execute - that's called a racing issue.

Each process will execute in order, but the relative timing between the two is not guaranteed.

So, it's possible that the order is

1. queue.put("Hello from main process")
2. data = queue.get() in sub process
3. queue.put("Hello from sub process")
4. data = queue.get() in main process

which works as expected. However, it could also be

1. queue.put("Hello from main process")
2. data = queue.get() in main process
3. data = queue.get() in sub process - stuck here forever because the queue is empty

The reason that it works on Linux but not Windows is probably fork is faster than spawn, but your code is fundamentally incorrect.

I would suggest searching for "dead-lock" or learning a bit more about concurrency issues, this is a classic one :)

[terryjreedy]

This appears to be a timing difference issue. On Windows, the main thread puts a string on the queue and removes it before the worker process has started, let alone get it, leaving the queue empty. So the blocking queue.get in the worker process blocks until the keyboard interrupt.

What is the output in linux? I guessing that you see "Subprocess started" either first or second, so that the worker is waiting to get the string as soon as it is put by the main process.

To me, this code is buggy. You should have two queues so only the worker can get the message from the main process, and vice versa. But note that the purpose of queue is, for instance, to let main send multiple jobs to multiple workers, each of which gets jobs when free, and collect results from multiple workers as they finish jobs. If there are only two processes, you can and perhaps should use a Pipe, which is bidirectional by default. Each process uses one end of a pipe to both send and 'recv' (receive).

import multiprocessing

def sub_process(sub):
    print("Sub process started")
    data = sub.recv()
    print(f"Sub process received '{data}' from main process")
    sub.send("Hello from sub process")
    print("Sub process finished")

if __name__ == "__main__":
    main, sub = multiprocessing.Pipe()
    p = multiprocessing.Process(target=sub_process, args=(sub,))
    p.start()
    main.send("Hello from main process")
    print("Main process sent data to sub process")
    data = main.recv()
    print(f"Main process received '{data}' from sub process")
    p.join()

outputs

Main process sent data to sub process
Sub process started
Sub process received 'Hello from main process' from main process
Sub process finished
Main process received 'Hello from sub process' from sub process