Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-implement multiprocessing for macOS #774

Open
jdesboeufs opened this issue Apr 6, 2023 · 0 comments
Open

Re-implement multiprocessing for macOS #774

jdesboeufs opened this issue Apr 6, 2023 · 0 comments

Comments

@jdesboeufs
Copy link
Member

Since Python 3.8, batch indexing is broken on macOS.

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/Users/jerome/.pyenv/versions/3.10.7/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/Users/jerome/.pyenv/versions/3.10.7/lib/python3.10/site-packages/addok/autocomplete.py", line 112, in index_ngram_keys
    pipe = DB.pipeline(transaction=False)
  File "/Users/jerome/.pyenv/versions/3.10.7/lib/python3.10/site-packages/addok/db.py", line 18, in __getattr__
    return getattr(self.instance, name)
AttributeError: 'NoneType' object has no attribute 'pipeline'
"""

After some investigation, I learned multiprocessing had a new behavior:

Changed in version 3.8: On macOS, the spawn start method is now the default. The fork start method should be considered unsafe as it can lead to crashes of the subprocess. See bpo-33725.

(source: https://docs.python.org/3/library/multiprocessing.html)

I temporarily fixed the problem disabling multiprocessing on macOS (af47d75).
Not critical since macOS is nearly always a development environment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant