Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support spawn multiprocessing mode on Mac OS and Windows #637

Closed
osma opened this issue Oct 31, 2022 · 0 comments · Fixed by #638
Closed

Support spawn multiprocessing mode on Mac OS and Windows #637

osma opened this issue Oct 31, 2022 · 0 comments · Fixed by #638
Milestone

Comments

@osma
Copy link
Member

osma commented Oct 31, 2022

Currently Annif performs parallel processing using the multiprocessing module for some functionality, for example parallel evaluation and preparing the training data for the NN ensemble. This relies (implicitly) on the fork model, which is the default on Linux, and has some nice benefits: models only need to be loaded once, and most of the memory can be shared between worker processes.

However, the fork mode is not available on Windows. On Mac OS, spawn is the default since Python 3.8 because there are problems with the fork mode. In practice, the parallel processing functionality will not work on Mac OS or Windows (unless Annif is run in a Docker container or a Linux VM). See for example this report on annif-users.

We could support the spawn mode as well (and possibly forkserver, although it's not available on Windows). This would be less efficient than fork, since each worker needs to load models separately and they will all require more memory. But it could still be useful on machines with many CPU cores and lots of RAM available.

@osma osma added this to the Long term milestone Oct 31, 2022
@osma osma closed this as completed in #638 Nov 23, 2022
@osma osma modified the milestones: Long term, 0.60 Dec 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant