Hanging mpire worker_pool.map_unordered

[fzonneveld]

When creating many processes which return large result, mpire will hang in collecting the results.
Analyzed is caused by crossing the system limit size for the os pipes, for linux defaults to 4K.

[sybrenjansen]

Would it be possible to provide a minimal working example where this happens? That would save me a lot of time.

And please share the MPIRE version you're using, thnx.

[fzonneveld]

import mpire    
    
def time_consuming_function(x):    
    print(os.getpid())    
    return os.urandom(1024 * 20)    
    
with mpire.WorkerPool(n_jobs=5) as pool: 
    results = pool.mapi_unordered(time_consuming_function, range(10), worker_lifespan=1)    
    print(len(results))    

Without the worker_lifespan it works fine.
Also pool.map has the same behavior.

[sybrenjansen]

Thnx for the example code. I can confirm that it indeed is hanging. I will have a look at this this week.

[sybrenjansen]

Already found the culprit. Will try to find an elegant solution now. Fix will be available in next release of mpire

[fzonneveld]

Super, thanks Op ma 21 feb. 2022 16:51 schreef Sybren Jansen ***@***.***>:

…

Already found the culprit. Will try to find an elegant solution now — Reply to this email directly, view it on GitHub <#32 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACH77I2SV4KRHYSFXT5EHV3U4JNO5ANCNFSM5N5ALP5A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[sybrenjansen]

Fixed in v2.3.4