-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Open
Labels
Description
Description
We almost have PyMC working natively under pyodide. But upon import we get an error importing multiprocessing which isn't included in pyodide:
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
Cell In[5], line 1
----> 1 import pymc as pm
2 pm.__version__
File /lib/python3.12/site-packages/pymc/__init__.py:72
70 from pymc.pytensorf import *
71 from pymc.sampling import *
---> 72 from pymc.smc import *
73 from pymc.stats import *
74 from pymc.step_methods import *
File /lib/python3.12/site-packages/pymc/smc/__init__.py:16
1 # Copyright 2024 The PyMC Developers
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
(...)
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
15 from pymc.smc.kernels import IMH, MH
---> 16 from pymc.smc.sampling import sample_smc
18 __all__ = ("sample_smc",)
File /lib/python3.12/site-packages/pymc/smc/sampling.py:21
18 import warnings
20 from collections import defaultdict
---> 21 from concurrent.futures import ProcessPoolExecutor, wait
22 from typing import Any
24 import cloudpickle
File /lib/python312.zip/concurrent/futures/__init__.py:44, in __getattr__(name)
41 global ProcessPoolExecutor, ThreadPoolExecutor
43 if name == 'ProcessPoolExecutor':
---> 44 from .process import ProcessPoolExecutor as pe
45 ProcessPoolExecutor = pe
46 return pe
File /lib/python312.zip/concurrent/futures/process.py:55
52 # This import is required to load the multiprocessing.connection submodule
53 # so that it can be accessed later as `mp.connection`
54 import multiprocessing.connection
---> 55 from multiprocessing.queues import Queue
56 import threading
57 import weakref
File /lib/python312.zip/multiprocessing/queues.py:23
19 import errno
21 from queue import Empty, Full
---> 23 import _multiprocessing
25 from . import connection
26 from . import context
ModuleNotFoundError: No module named '_multiprocessing'
If we made that optional, it will work out of the box.
Activity
ricardoV94 commentedon Oct 2, 2024
I don't think it makes sense to go out of our way to make a standard python lib import optional. It's an anti-pattern to import things inside functions.
twiecki commentedon Oct 2, 2024
I agree it's an anti-pattern, but I think it certainly makes sense here because with a pretty simple change we enable PyMC in the browser. So I think the added tech debt is worth it. We can also make it optional if we're running on pyodide.
ricardoV94 commentedon Oct 2, 2024
How does that look like?
twiecki commentedon Oct 2, 2024
ricardoV94 commentedon Oct 2, 2024
That's even uglier
twiecki commentedon Oct 2, 2024
So what do we do?
ricardoV94 commentedon Oct 2, 2024
Local import or nothing. Should an issue be open with pyodide as well?
twiecki commentedon Oct 2, 2024
OK, I don't mind which way. What issue would that be?
[-]Make multiprocessing import optional/local to support pyodide[/-][+]Make multiprocessing import local to support pyodide[/+]ricardoV94 commentedon Oct 2, 2024
Shouldn't pyodide support the python standard library?
twiecki commentedon Oct 2, 2024
I don't think it's an oversight, they just haven't figured out multiprocessing yet.
adithyalaks commentedon Nov 5, 2024
Looking at this from the PyData NYC sprint. I don't really see how we could unblock this given that
multiprocessing
is used in various other modules inPyMC
. Would we make the import local to all of the modules that use it??twiecki commentedon Nov 6, 2024
Where else is it being used? I guess that'd be the only way then, depends in how many modules we use it.
AtulBoyal commentedon Mar 14, 2025
Hi, I’d love to work on this issue as my first contribution. Is it still available?
twiecki commentedon Mar 14, 2025
Yes!
Fix pymc-devs#7519: Move multiprocessing import to local scope in run…
AtulBoyal commentedon Mar 25, 2025
Hi @twiecki @ricardoV94,
Updated PR #7736 to fix #7519 by moving
multiprocessing
import torun_chains
. Also enhancedtest_mcbackend.py
.Local tests failed (
CompileError
), Colab passedtest_censored.py
. Fixing MinGW next.Thanks,
Atul Boyal
AtulBoyal commentedon Mar 25, 2025
Hi @twiecki @ricardoV94,
I’m 17 until June 30, missing GSoC 2025 registration (ends April 8). PR #7736 is in, and I fixed
test_censored.py
locally with 64-bit MinGW (all tests pass now). I’d love to keep working on PyTensor compilation fixes for Windows—any chance for unofficial mentorship this summer?Thanks,
Atul Boyal
(AnInnovativeCoder)
koaning commentedon May 14, 2025
I have found a bit of a hack on my marimo WASM notebook.
This seems to get past the multiprocessing bug, but I seem to hit another issue down the line.
This yields:
Figured I'd mention it in case it might be something that's also caused by pyodide.
twiecki commentedon May 18, 2025
Yeah, nutpie won't work on wasm.