concurrent.futures.ProcessPoolExecutor state=finished raised error

[maggyero]

BPO	37294
Nosy	@brianquinlan, @pitrou, @ned-deily, @asvetlov, @methane, @ambv, @serhiy-storchaka, @maggyero, @aeros, @danil Zherebtsov
Superseder	bpo-37297: function changed when pickle bound method object

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-06-28.18:50:54.545>
created_at = <Date 2019-06-15.17:49:58.796>
labels = ['3.7', 'expert-2to3', 'type-crash']
title = 'concurrent.futures.ProcessPoolExecutor state=finished raised error'
updated_at = <Date 2020-10-01.07:38:09.931>
user = 'https://github.com/maggyero'

bugs.python.org fields:

activity = <Date 2020-10-01.07:38:09.931>
actor = 'DanilZ'
assignee = 'none'
closed = True
closed_date = <Date 2019-06-28.18:50:54.545>
closer = 'bquinlan'
components = ['2to3 (2.x to 3.x conversion tool)']
creation = <Date 2019-06-15.17:49:58.796>
creator = 'maggyero'
dependencies = []
files = []
hgrepos = []
issue_num = 37294
keywords = []
message_count = 9.0
messages = ['345709', '345714', '345716', '345717', '345777', '377610', '377727', '377738', '377739']
nosy_count = 10.0
nosy_names = ['bquinlan', 'pitrou', 'ned.deily', 'asvetlov', 'methane', 'lukasz.langa', 'serhiy.storchaka', 'maggyero', 'aeros', 'DanilZ']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '37297'
type = 'crash'
url = 'https://bugs.python.org/issue37294'
versions = ['Python 3.7']

[maggyero]

The following code hangs forever instead of printing "called" 10 times:

    from concurrent.futures import ProcessPoolExecutor
    
    class A:
        def f(self):
            print("called")
    
    class B(A):
        def f(self):
            executor = ProcessPoolExecutor(max_workers=2)
            futures = [executor.submit(super(B, self).f)
                       for _ in range(10)]
    
    if __name__ == "__main__":
        B().f()

The same code with super(B, self) replaced with super() raises the following error:

TypeError: super(type, obj): obj must be an instance or subtype of type

However, replacing ProcessPoolExecutor with ThreadPoolExecutor works as expected, but only with super(B, self) (with super() it still raises the same error).

[asvetlov]

The use case is weird.
I don't think we need to do something with the issue.

[maggyero]

@andrew Svetlov

Well this was just an example for illustrating the issue. In real code I need this to create ThreadPoolMixin and ProcessPoolMixin classes (similar to the socketserver.ThreadingMixIn and socketserver.ForkingMixIn classes in the standard library, but using a thread/process pool with a fixed size instead of creating a new thread/process per request) for mixing with my server classes. But because of this ProcessPoolExecutor issue, I cannot use my ProcessPoolMixin class but only my ThreadPoolMixin currently.

The fact is one cannot submit a parent method call to a ProcessPoolExecutor.

This code snippet prints the exception raised by the call:

from concurrent.futures import ProcessPoolExecutor

class A:
    def f(self):
        print("called")

class B(A):
    def f(self):
        executor = ProcessPoolExecutor(max_workers=2)
        print(executor.submit(super().f).exception())

if __name__ == "__main__":
    B().f()

It prints this:

[Errno 24] Too many open files
None
None
None
…

[maggyero]

And like the concurrent.futures module for concurrent.futures.ProcessPoolExecutor but not for concurrent.futures.ThreadPoolExecutor (see above), the multiprocessing.pool module seems also affected by a similar problem for multiprocessing.pool.Pool (process pools) but not for multiprocessing.pool.ThreadPool (thread pools).

Indeed the following code:

    import multiprocessing.pool
    
    class A:
        def f(self):
            print("called")
    
    class B(A):
        def f(self):
            pool = multiprocessing.pool.Pool(2)
            pool.apply(super().f)
    
    if __name__ == "__main__":
        B().f()

raises the following exception:

AssertionError: daemonic processes are not allowed to have children

[maggyero]

George Xie found the root cause of this issue (a bug in the function method_reduce in cpython/Objects/classobject.c):

https://stackoverflow.com/questions/56609847/why-do-concurrent-futures-processpoolexecutor-and-multiprocessing-pool-pool-fail/56614748#56614748

and he filed the specific bug here:

https://bugs.python.org/issue37297

The issue looks serious as it creates an infinite recursion that crashed my laptop several times today.

[DanilZherebtsov]

After executing a single task inside a process the result is returned with state=finished raised error.

Error happens when trying to load a big dataset (over 5 GB). Otherwise the same dataset reduced to a smaller nrows executes and returns from result() without errors.

with concurrent.futures.ProcessPoolExecutor(max_workers = 1) as executor:
    results = executor.submit(pd.read_csv, path)

data = results.result()

[aeros]

DanilZ, could you take a look at the superseding issue (https://bugs.python.org/issue37297) and see if your exception raised within the job is the same?

If it's not, I would suggest opening a separate issue (and linking to it in a comment here), as I don't think it's necessarily related to this one. "state=finished raised error" doesn't indicate the specific exception that occurred. A good format for the name would be something along the lines of:

"ProcessPoolExecutor.submit() <specific exception name here> while reading large object (4GB)"

It'd also be helpful in the separate issue to paste the full exception stack trace, specify OS, and multiprocessing start method used (spawn, fork, or forkserver). This is necessary to know for replicating the issue on our end.

In the meantime, I workaround I would suggest trying would be to use the *chunksize* parameter (or *Iterator*) in pandas.read_csv(), and split it across several jobs (at least 4+, more if you have additional cores) instead of within a single one. It'd also be generally helpful to see if that alleviates the problem, as it could possibly indicate an issue with running out of memory when the dataframe is converted to pickle format (which often increases the total size) within the process associated with the job.