Skip to content

Make multiprocessing import local to support pyodide #7519

@twiecki

Description

@twiecki
Member

Description

We almost have PyMC working natively under pyodide. But upon import we get an error importing multiprocessing which isn't included in pyodide:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[5], line 1
----> 1 import pymc as pm
      2 pm.__version__

File /lib/python3.12/site-packages/pymc/__init__.py:72
     70 from pymc.pytensorf import *
     71 from pymc.sampling import *
---> 72 from pymc.smc import *
     73 from pymc.stats import *
     74 from pymc.step_methods import *

File /lib/python3.12/site-packages/pymc/smc/__init__.py:16
      1 #   Copyright 2024 The PyMC Developers
      2 #
      3 #   Licensed under the Apache License, Version 2.0 (the "License");
   (...)
     12 #   See the License for the specific language governing permissions and
     13 #   limitations under the License.
     15 from pymc.smc.kernels import IMH, MH
---> 16 from pymc.smc.sampling import sample_smc
     18 __all__ = ("sample_smc",)

File /lib/python3.12/site-packages/pymc/smc/sampling.py:21
     18 import warnings
     20 from collections import defaultdict
---> 21 from concurrent.futures import ProcessPoolExecutor, wait
     22 from typing import Any
     24 import cloudpickle

File /lib/python312.zip/concurrent/futures/__init__.py:44, in __getattr__(name)
     41 global ProcessPoolExecutor, ThreadPoolExecutor
     43 if name == 'ProcessPoolExecutor':
---> 44     from .process import ProcessPoolExecutor as pe
     45     ProcessPoolExecutor = pe
     46     return pe

File /lib/python312.zip/concurrent/futures/process.py:55
     52 # This import is required to load the multiprocessing.connection submodule
     53 # so that it can be accessed later as `mp.connection`
     54 import multiprocessing.connection
---> 55 from multiprocessing.queues import Queue
     56 import threading
     57 import weakref

File /lib/python312.zip/multiprocessing/queues.py:23
     19 import errno
     21 from queue import Empty, Full
---> 23 import _multiprocessing
     25 from . import connection
     26 from . import context

ModuleNotFoundError: No module named '_multiprocessing'

If we made that optional, it will work out of the box.

Activity

ricardoV94

ricardoV94 commented on Oct 2, 2024

@ricardoV94
Member

I don't think it makes sense to go out of our way to make a standard python lib import optional. It's an anti-pattern to import things inside functions.

twiecki

twiecki commented on Oct 2, 2024

@twiecki
MemberAuthor

I agree it's an anti-pattern, but I think it certainly makes sense here because with a pretty simple change we enable PyMC in the browser. So I think the added tech debt is worth it. We can also make it optional if we're running on pyodide.

ricardoV94

ricardoV94 commented on Oct 2, 2024

@ricardoV94
Member

We can also make it optional if we're running on pyodide.

How does that look like?

twiecki

twiecki commented on Oct 2, 2024

@twiecki
MemberAuthor
ricardoV94

ricardoV94 commented on Oct 2, 2024

@ricardoV94
Member

That's even uglier

twiecki

twiecki commented on Oct 2, 2024

@twiecki
MemberAuthor

So what do we do?

ricardoV94

ricardoV94 commented on Oct 2, 2024

@ricardoV94
Member

Local import or nothing. Should an issue be open with pyodide as well?

twiecki

twiecki commented on Oct 2, 2024

@twiecki
MemberAuthor

OK, I don't mind which way. What issue would that be?

changed the title [-]Make multiprocessing import optional/local to support pyodide[/-] [+]Make multiprocessing import local to support pyodide[/+] on Oct 2, 2024
ricardoV94

ricardoV94 commented on Oct 2, 2024

@ricardoV94
Member

OK, I don't mind which way. What issue would that be?

Shouldn't pyodide support the python standard library?

twiecki

twiecki commented on Oct 2, 2024

@twiecki
MemberAuthor

Shouldn't pyodide support the python standard library?

I don't think it's an oversight, they just haven't figured out multiprocessing yet.

adithyalaks

adithyalaks commented on Nov 5, 2024

@adithyalaks

Looking at this from the PyData NYC sprint. I don't really see how we could unblock this given that multiprocessing is used in various other modules in PyMC. Would we make the import local to all of the modules that use it??

twiecki

twiecki commented on Nov 6, 2024

@twiecki
MemberAuthor

Where else is it being used? I guess that'd be the only way then, depends in how many modules we use it.

AtulBoyal

AtulBoyal commented on Mar 14, 2025

@AtulBoyal

Hi, I’d love to work on this issue as my first contribution. Is it still available?

twiecki

twiecki commented on Mar 14, 2025

@twiecki
MemberAuthor

Yes!

added a commit that references this issue on Mar 24, 2025
e42a401
AtulBoyal

AtulBoyal commented on Mar 25, 2025

@AtulBoyal

Hi @twiecki @ricardoV94,

Updated PR #7736 to fix #7519 by moving multiprocessing import to run_chains. Also enhanced test_mcbackend.py.

Local tests failed (CompileError), Colab passed test_censored.py. Fixing MinGW next.

Thanks,
Atul Boyal

AtulBoyal

AtulBoyal commented on Mar 25, 2025

@AtulBoyal

Hi @twiecki @ricardoV94,

I’m 17 until June 30, missing GSoC 2025 registration (ends April 8). PR #7736 is in, and I fixed test_censored.py locally with 64-bit MinGW (all tests pass now). I’d love to keep working on PyTensor compilation fixes for Windows—any chance for unofficial mentorship this summer?

Thanks,
Atul Boyal
(AnInnovativeCoder)

koaning

koaning commented on May 14, 2025

@koaning

I have found a bit of a hack on my marimo WASM notebook.

import sys
sys.modules['_multiprocessing'] = object
from multiprocessing.pool import ThreadPool

This seems to get past the multiprocessing bug, but I seem to hit another issue down the line.

with Model() as model:  # model specifications in PyMC are wrapped in a with-statement
    # Define priors
    sigma = HalfCauchy("sigma", beta=10)
    intercept = Normal("Intercept", 0, sigma=20)
    slope = Normal("slope", 0, sigma=20)

    # Define likelihood
    likelihood = Normal("y", mu=intercept + slope * x, sigma=sigma, observed=y)

    # Inference!
    # draw 3000 posterior samples using NUTS sampling
    idata = sample(3000, cores=1, nuts_sampler="nutpie")

This yields:

Traceback (most recent call last):
  File "/lib/python3.12/site-packages/marimo/_runtime/executor.py", line 141, in execute_cell
    exec(cell.body, glbls)
  Cell marimo://notebook.py#cell=cell-7, line 12, in <module>
    idata = sample(3000, cores=1, nuts_sampler="nutpie")
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/pymc/sampling/mcmc.py", line 789, in sample
    provided_steps, selected_steps = assign_step_methods(model, step, methods=pm.STEP_METHODS)
                                                                              ^^^^^^^^^^^^^^^
AttributeError: module 'pymc' has no attribute 'STEP_METHODS'

Figured I'd mention it in case it might be something that's also caused by pyodide.

twiecki

twiecki commented on May 18, 2025

@twiecki
MemberAuthor

Yeah, nutpie won't work on wasm.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Participants

      @fonnesbeck@twiecki@koaning@adithyalaks@ricardoV94

      Issue actions

        Make multiprocessing import local to support pyodide · Issue #7519 · pymc-devs/pymc