resource_tracker: There appear to be 32 leaked semaphore objects to clean up at shutdown

[sailxjx]

I've got this message by resource tracker when I use spawn mode on my mac, the code is really simple:

import time
from mpire import WorkerPool


def global_runner(*args):
    print("I am Running", args)
    time.sleep(0.3)
    return


if __name__ == "__main__":
    params_group = [1, 2]
    with WorkerPool(n_jobs=2, start_method="spawn") as pool:
        pool.map(global_runner, params_group)

Can anyone help me solve this problem?

[sailxjx]

The version of python is 3.9.7

[sybrenjansen]

Hmm.. Unfortunately I don't have a mac to be able to debug this. When testing this on linux using python 3.9.7 your script works just fine and doesn't give the warnings. So it's probably mac specific.

However, it is only a warning at the end of the script, and a warning like this can be safely ignored. I.e., your code should still work fine.

If someone else wants to do some debugging, please do so. The only semaphores being used in mpire are within JoinableQueues in mpire/comms.py.

[macintacos]

Don't ask me how, but I seem to be consistently getting this when using a WorkerPool with progress_bar=True. When I remove the progress bar (and remove start_method="spawn") the issue goes away.

[jonashaag]

There is definitely a bug here that leaks semaphores. It causes macOS to run out of file descriptors.

[sybrenjansen]

Bug has been found. Mac support will be added soon