multiprocessing.Pool hangs on certain custom Exceptions

[djm4686]

Bug report

Reproducing:

import multiprocessing


class MyExc(Exception):
    def __init__(self, stuff):
        super().__init__(stuff.two)


class Stuff:
    def __init__(self):
        self.one = 1
        self.two = 2


def run(_):
    raise MyExc(Stuff())


if __name__ == "__main__":
    with multiprocessing.Pool(3) as pool:
        pool.map(run, [1])

This will cause python to hang indefinitely. The cause is the custom exception initialization.

This will throw an exception in the subprocess and will not fail out
AttributeError: 'int' object has no attribute 'two'

Your environment

CPython 3.10.8
Mac OSX Ventura 13.2

[gaogaotiantian]

The trigger of this problem is pickle. Exception classes assume that it can restore itself when pickling with the arguments given, but that's not the case in this example. One could argue that this is not the correct way to use Exception.

The exception did not happen in the subprocess, it happened in the main process, _handle_results thread. When the thread got a pickled result from the subprocess, it thought it could restore the Exception, but it failed because the custom Exception does not follow the initialization rules.

I guess that's a question for the multiprocessing lib owner - should we monitor the potential pickling error on _handle_results and bail out the process when it happens?

[peterallenwebb]

This is a very surprising behavior and caused recent versions of dbt-core to hang on thread exceptions. Would be great to see this fixed if it is possible.