multiprocessing.Queue does pickling in a worker thread, thus pickling errors don't generate helpful tracebacks.

[remdragon]

Bug report

Bug description:

I recently got the following traceback in my code:

Traceback (most recent call last):
  File "multiprocessing\queues.py", line 244, in _feed
  File "multiprocessing\reduction.py", line 51, in dumps
TypeError: cannot pickle '_thread.RLock' object

Unfortunately, there's nothing about this traceback that is useful or helpful to me. I started digging through the logic and it appears that multiprocessing.Queue has a worker thread where it does the pickling. The unfortunate side-effect of this is that if there a pickling error, it doesn't happen in a useful context. If the pickling could happen before the data is handed off to the worker thread, then a much more useful traceback could be generated.

CPython versions tested on:

3.11

Operating systems tested on:

Windows

[gaogaotiantian]

I don't believe this is a bug - it's not against any documented behavior.

There is a reason to put the pickling in another thread - to unblock the main thread. The main thread do not need to wait for the object to be pickled, it was dealt in a worker thread. I don't think "having a better trackback" (even though I agree that would be nice) is strong enough to change the behavior.

[remdragon]

I'm afraid I don't understand the performance argument given the GIL, but I guess do what you will, thanks.

[gaogaotiantian]

I'm afraid I don't understand the performance argument given the GIL, but I guess do what you will, thanks.

Ah, I might've been wrong on that statement - yes because pickle normally use the C implementation so it won't yield from the pickling work, and the main thread is probably stuck when the worker thread is dumping the data.

However, there's another concern to "pickle it ahead" - memory usage. The pickled object could be large (especially compared to the current object stored in a list which is essentially free). It's a queue so we could potentially have a lot of objects in it. If all of them are pickled ahead, then it'll cost a significant amount of memory.

Where as in the current implementation, only one pickled data exists at the same time for a queue.

Improving debuggability when something goes wrong is a great path to go, but we should always consider the actual impact on the program on the happy path.

[remdragon]

I think those are fair concerns. The challenge is that by doing the work in a thread and allowing the calling thread to proceed, by the time an error happens our context to throw an exception in is ancient history.

I wonder if it would be possible to add a feature to multiprocessing.Queue to allow a user to optin to early pickling when they're trying to hunt down an issue?

[dimaqq]

A simple fix would be to stringify the exception.

Another option is to record the exception and re-raise in caller's thread, but I think that implies keeping a bunch of stack frames referenced, and I think that's possibly frowned upon in cpython stdlib...

[serhiy-storchaka]

It is a duplicate of #84376.