Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No Sub-Packages of scipy (or numpy) can be Imported #619

Closed
michaelweinold opened this issue Aug 25, 2023 · 22 comments
Closed

No Sub-Packages of scipy (or numpy) can be Imported #619

michaelweinold opened this issue Aug 25, 2023 · 22 comments
Labels
bug Something isn't working

Comments

@michaelweinold
Copy link
Contributor

michaelweinold commented Aug 25, 2023

If I try to import a subpackage from scipy:

import scipy.interpolate

in a JupyterLite environment built with

jupyter_server=2.7.1
jupyterlite-core=0.1.2
jupyterlite-xeus-python=0.9.4

I get the error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[15], line 1
----> 1 import scipy.interpolate

File /lib/python3.10/site-packages/scipy/interpolate/__init__.py:166
      1 """
      2 ========================================
      3 Interpolation (:mod:`scipy.interpolate`)
   (...)
    164 (should not be used in new code).
    165 """
--> 166 from ._interpolate import *
    167 from ._fitpack_py import *
    169 # New interface to fitpack library:

File /lib/python3.10/site-packages/scipy/interpolate/_interpolate.py:21
     19 from . import _ppoly
     20 from ._fitpack2 import RectBivariateSpline
---> 21 from .interpnd import _ndim_coords_from_arrays
     22 from ._bsplines import make_interp_spline, BSpline
     25 def lagrange(x, w):

File interpnd.pyx:1, in init scipy.interpolate.interpnd()

AttributeError: module 'qhull' has no attribute '__pyx_capi__'

This affects our attempt to run Brightway in JupyterLite, since we are using functions from scipy.interpolate.

This works fine in the 0.23.4 version of Pyodide:

A WebAssembly-powered Python kernel backed by Pyodide
> import scipy.interpolate
@martinRenou
Copy link
Contributor

Thanks for opening an issue, I'm looking into it

...this seems to affect numpy as well. I didn't test for any other packages

I am unable to replicate this, is there an import from Numpy in particular that you see failing?

@michaelweinold
Copy link
Contributor Author

import numpy
def import_all_subpackages():
    error_subpackages = []
    for subpackage in numpy.__all__:
        try:
            __import__(f"numpy.{subpackage}")
        except ImportError:
            error_subpackages.append(subpackage)
    if error_subpackages:
        print("Error importing the following subpackages:")
        print(error_subpackages)
    else:
        print("All subpackages imported successfully!")
import_all_subpackages()

Run in my test repo results in:

Screenshot 2023-08-25 at 09 37 26

@martinRenou
Copy link
Contributor

Nice test 👍🏽 That's a one we should add in the Numpy recipe

@martinRenou
Copy link
Contributor

I read a bit too fast.

Those "subpackages" it cannot import are not subpackages. Those are attributes.

e.g. there is no numpy.array module that you can import (whether it's lite or classic Python), though it is listed in your "subpackages"

@martinRenou
Copy link
Contributor

I guess that's what you wanted to do:

import pkgutil
import numpy

for submodule in pkgutil.iter_modules(numpy.__path__):
    try:
        __import__(f"numpy.{submodule.name}")
    except ImportError:
        print(f"failed to import {submodule.name}")

@martinRenou
Copy link
Contributor

That's weird, I can import scipy.interpolate on this deployment: https://martinrenou.github.io/test-xeus-python

Using the same package versions as you do.

@michaelweinold
Copy link
Contributor Author

...I can't confirm. On your JupyterLite site, I can't even import scipy:

/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.17.3 and <1.25.0 is required for this version of SciPy (detected version 1.25.2
  warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
Screenshot 2023-08-25 at 10 29 12

This is on Chrome Version 116.0.5845.96 (Official Build) (arm64) under macOS 13.5.1 (22G90).

@martinRenou
Copy link
Contributor

martinRenou commented Aug 25, 2023

This is just a warning, in your screenshot, scipy was imported correctly.

EDIT In order to get rid of the warning, we need to update the scipy recipe to use the latest Scipy version, a PR is opened for this but it needs fixing

@martinRenou
Copy link
Contributor

It looks like the following results in AttributeError: module 'qhull' has no attribute '__pyx_capi__'

  + arrow-python                     11.0.0  py310he6e8567_3    repo.mamba.pm/emscripten-forge/emscripten-32      33MB
  + asttokens                         2.2.1  pyhd8ed1ab_0       conda-forge/noarch                                28kB
  + backcall                          0.2.0  pyh9f0ad1d_0       repo.mamba.pm/conda-forge/noarch                  14kB
  + backports                           1.0  pyhd8ed1ab_3       conda-forge/noarch                                 6kB
  + backports.functools_lru_cache     1.6.5  pyhd8ed1ab_0       conda-forge/noarch                                12kB
  + clapack                           3.2.1  h29e2456_4         repo.mamba.pm/emscripten-forge/emscripten-32       1MB
  + cloudpickle                       2.2.1  pyhd8ed1ab_0       conda-forge/noarch                                28kB
  + cycler                           0.11.0  pyhd8ed1ab_0       repo.mamba.pm/conda-forge/noarch                  10kB
  + decorator                         5.1.1  pyhd8ed1ab_0       repo.mamba.pm/conda-forge/noarch                  12kB
  + emscripten-abi                   3.1.27  hb0f4dca_6         repo.mamba.pm/emscripten-forge/noarch              4kB
  + executing                         1.2.0  pyhd8ed1ab_0       repo.mamba.pm/conda-forge/noarch                  25kB
  + fonttools                        4.39.4  py310h6d2fff6_0    repo.mamba.pm/emscripten-forge/emscripten-32       2MB
  + fsspec                         2023.6.0  pyh1a96a4e_0       conda-forge/noarch                               118kB
  + ipython                          8.14.0  py310h8bed8af_1    repo.mamba.pm/emscripten-forge/emscripten-32       1MB
  + jedi                             0.19.0  pyhd8ed1ab_0       conda-forge/noarch                               845kB
  + kiwisolver                        1.3.2  hbf4eec8_3         repo.mamba.pm/emscripten-forge/emscripten-32      32kB
  + libffi                            3.4.2  hce8ced0_0         repo.mamba.pm/emscripten-forge/emscripten-32      16kB
  + matplotlib                        3.5.2  py310h74ff0a5_0    repo.mamba.pm/emscripten-forge/emscripten-32       2MB
  + matplotlib-base                   3.5.2  h74ff0a5_0         repo.mamba.pm/emscripten-forge/emscripten-32      10MB
  + matplotlib-inline                 0.1.6  pyhd8ed1ab_0       repo.mamba.pm/conda-forge/noarch                  12kB
  + numpy                            1.25.2  py310h6d2fff6_0    repo.mamba.pm/emscripten-forge/emscripten-32       7MB
  + packaging                          23.1  pyhd8ed1ab_0       conda-forge/noarch                                46kB
  + pandas                            1.4.3  hbe3984b_0         repo.mamba.pm/emscripten-forge/emscripten-32      10MB
  + parso                             0.8.3  pyhd8ed1ab_0       repo.mamba.pm/conda-forge/noarch                  71kB
  + pexpect                           4.8.0  pyh1a96a4e_2       repo.mamba.pm/conda-forge/noarch                  49kB
  + pickleshare                       0.7.5  py_1003            repo.mamba.pm/conda-forge/noarch                   9kB
  + pillow                            9.1.0  h5554403_1         repo.mamba.pm/emscripten-forge/emscripten-32     926kB
  + pip                              23.2.1  pyhd8ed1ab_0       conda-forge/noarch                                 1MB
  + prompt-toolkit                   3.0.39  pyha770c72_0       conda-forge/noarch                               269kB
  + ptyprocess                        0.7.0  pyhd3deb0d_0       repo.mamba.pm/conda-forge/noarch                  17kB
  + pure_eval                         0.2.2  pyhd8ed1ab_0       repo.mamba.pm/conda-forge/noarch                  15kB
  + pycparser                          2.21  pyhd8ed1ab_0       repo.mamba.pm/conda-forge/noarch                 103kB
  + pygments                         2.16.1  pyhd8ed1ab_0       conda-forge/noarch                               8[53](https://github.com/martinRenou/test-xeus-python/actions/runs/5974150785/job/16207746765#step:5:54)kB
  + pyparsing                         3.1.1  pyhd8ed1ab_0       conda-forge/noarch                                90kB
  + python                           3.10.2  h_hash_26_cpython  repo.mamba.pm/emscripten-forge/emscripten-32       8MB
  + python-dateutil                   2.8.2  pyhd8ed1ab_0       repo.mamba.pm/conda-forge/noarch                 246kB
  + pytz                             2023.3  pyhd8ed1ab_0       conda-forge/noarch                               187kB
  + requests-wasm-polyfill            0.3.0  h8bed8af_0         repo.mamba.pm/emscripten-forge/emscripten-32      11kB
  + scipy                             1.8.1  py310h74ff0a5_4    repo.mamba.pm/emscripten-forge/emscripten-32      19MB
  + setuptools                       64.0.0  py310h8bed8af_0    repo.mamba.pm/emscripten-forge/emscripten-32     776kB
  + six                              1.16.0  pyh6c4a22f_0       repo.mamba.pm/conda-forge/noarch                  14kB
  + stack_data                        0.6.2  pyhd8ed1ab_0       conda-forge/noarch                                26kB
  + traitlets                         5.9.0  pyhd8ed1ab_0       conda-forge/noarch                                98kB
  + wcwidth                           0.2.6  pyhd8ed1ab_0       conda-forge/noarch                                29kB
  + wheel                            0.41.2  pyhd8ed1ab_0       conda-forge/noarch                                [57](https://github.com/martinRenou/test-xeus-python/actions/runs/5974150785/job/16207746765#step:5:58)kB
  + xeus-lite                         1.0.1  hf0acc6b_0         repo.mamba.pm/emscripten-forge/emscripten-32      69kB
  + xeus-python                      0.15.9  h4d17c[68](https://github.com/martinRenou/test-xeus-python/actions/runs/5974150785/job/16207746765#step:5:69)_1         repo.mamba.pm/emscripten-forge/emscripten-32       3MB
  + xeus-python-shell                 0.6.1  pyhd8ed1ab_0       conda-forge/noarch                                 7kB
  + xeus-python-shell-raw             0.6.1  pyhd8ed1ab_0       conda-forge/noarch                                12kB

But the following works:

  + asttokens                        2.2.1  pyhd8ed1ab_0       conda-forge/noarch                                28kB
  + backcall                         0.2.0  pyh9f0ad1d_0       repo.mamba.pm/conda-forge/noarch                  14kB
  + backports                          1.0  pyhd8ed1ab_3       conda-forge/noarch                                 6kB
  + backports.functools_lru_cache    1.6.5  pyhd8ed1ab_0       conda-forge/noarch                                12kB
  + clapack                          3.2.1  h29e2456_4         repo.mamba.pm/emscripten-forge/emscripten-32       1MB
  + decorator                        5.1.1  pyhd8ed1ab_0       repo.mamba.pm/conda-forge/noarch                  12kB
  + emscripten-abi                  3.1.27  hb0f4dca_6         repo.mamba.pm/emscripten-forge/noarch              4kB
  + executing                        1.2.0  pyhd8ed1ab_0       repo.mamba.pm/conda-forge/noarch                  25kB
  + ipython                         8.14.0  py310h8bed8af_1    repo.mamba.pm/emscripten-forge/emscripten-32       1MB
  + jedi                            0.19.0  pyhd8ed1ab_0       conda-forge/noarch                               845kB
  + matplotlib-inline                0.1.6  pyhd8ed1ab_0       repo.mamba.pm/conda-forge/noarch                  12kB
  + numpy                           1.25.2  py310h6d2fff6_0    repo.mamba.pm/emscripten-forge/emscripten-32       7MB
  + parso                            0.8.3  pyhd8ed1ab_0       repo.mamba.pm/conda-forge/noarch                  71kB
  + pexpect                          4.8.0  pyh1a96a4e_2       repo.mamba.pm/conda-forge/noarch                  [49](https://github.com/martinRenou/test-xeus-python/actions/runs/5973218114/job/16205057862#step:5:50)kB
  + pickleshare                      0.7.5  py_1003            repo.mamba.pm/conda-forge/noarch                   9kB
  + pip                             23.2.1  pyhd8ed1ab_0       conda-forge/noarch                                 1MB
  + prompt-toolkit                  3.0.39  pyha770c72_0       conda-forge/noarch                               269kB
  + ptyprocess                       0.7.0  pyhd3deb0d_0       repo.mamba.pm/conda-forge/noarch                  17kB
  + pure_eval                        0.2.2  pyhd8ed1ab_0       repo.mamba.pm/conda-forge/noarch                  15kB
  + pygments                        2.16.1  pyhd8ed1ab_0       conda-forge/noarch                               8[53](https://github.com/martinRenou/test-xeus-python/actions/runs/5973218114/job/16205057862#step:5:54)kB
  + pyparsing                        3.1.1  pyhd8ed1ab_0       conda-forge/noarch                                90kB
  + python                          3.10.2  h_hash_26_cpython  repo.mamba.pm/emscripten-forge/emscripten-32       8MB
  + requests-wasm-polyfill           0.3.0  h8bed8af_0         repo.mamba.pm/emscripten-forge/emscripten-32      11kB
  + scipy                            1.8.1  py310h74ff0a5_4    repo.mamba.pm/emscripten-forge/emscripten-32      19MB
  + setuptools                      64.0.0  py310h8bed8af_0    repo.mamba.pm/emscripten-forge/emscripten-32     776kB
  + six                             1.16.0  pyh6c4a22f_0       repo.mamba.pm/conda-forge/noarch                  14kB
  + stack_data                       0.6.2  pyhd8ed1ab_0       conda-forge/noarch                                26kB
  + traitlets                        5.9.0  pyhd8ed1ab_0       conda-forge/noarch                                98kB
  + wcwidth                          0.2.6  pyhd8ed1ab_0       conda-forge/noarch                                29kB
  + wheel                           0.41.2  pyhd8ed1ab_0       conda-forge/noarch                                [57](https://github.com/martinRenou/test-xeus-python/actions/runs/5973218114/job/16205057862#step:5:58)kB
  + xeus-lite                        1.0.1  hf0acc6b_0         repo.mamba.pm/emscripten-forge/emscripten-32      69kB
  + xeus-python                     0.15.9  h4d17c[68](https://github.com/martinRenou/test-xeus-python/actions/runs/5973218114/job/16205057862#step:5:69)_1         repo.mamba.pm/emscripten-forge/emscripten-32       3MB
  + xeus-python-shell                0.6.1  pyhd8ed1ab_0       conda-forge/noarch                                 7kB
  + xeus-python-shell-raw            0.6.1  pyhd8ed1ab_0       conda-forge/noarch                                12kB

@michaelweinold
Copy link
Contributor Author

Ok, so from my side:

That's weird, I can import scipy.interpolate on this deployment: https://martinrenou.github.io/test-xeus-python
Using the same package versions as you do.

I just tried this - it works for me as well.

Unfortunately, I cannot even reproduce the original error in our environment (https://brightway-lca.github.io/brightway-hub/):

Screenshot 2023-08-25 at 11 32 53

The kernel is always 'busy', but won't execute a single cell. I tried:

  1. Clearing the Indexed DB of the JupyterLite site
  2. Rebooting

I am not sure if this should be a separate issue? I couldn't find any relevant issues in the JupyterLite repo.

I'm not sure how to proceed - I can't even get the JupyterLite site to even execute a cell with 1+1 reliably, much less reproduce my original error.

@martinRenou
Copy link
Contributor

This looks very much like a cache issue. Can you try running JupyterLite in a new private window?

@martinRenou
Copy link
Contributor

After some more testing. It looks like the import scipy.interpolate failure comes as soon as you install matplotlib alongside scipy.

It may be because matplotlib and scipy are vendoring a different version of qhull?

@michaelweinold
Copy link
Contributor Author

This looks very much like a cache issue. Can you try running JupyterLite in a new private window?

This worked - now I can at least reproduce the original issue (thank you 🙏).

@martinRenou
Copy link
Contributor

Bear with me, we'll get there 😄

@michaelweinold
Copy link
Contributor Author

Hello, hello! Can we help with this issue in any way, @martinRenou?

@martinRenou
Copy link
Contributor

I guess the root of the issue is this:

matplotlib and scipy are vendoring a different version of qhull?

I'm hoping that rebuilding matplotlib and scipy on emscripten-forge could fix this, but both recipes fail to build as of today.

We basically need the CI on #527 to be green, same for #470

@michaelweinold
Copy link
Contributor Author

...as far as I can see, the error messages (eg. here for #527) are rather non-descript. However, the configuration in conda_build_config.yaml already has two variables defining verbosity set to 1. Is there a way to get some more information from the build?

@michaelweinold
Copy link
Contributor Author

Hey, @martinRenou - while we're still waiting for pyodide 0.24.0 to drop, could you update us on the scipy/matplotlib issue? Do you have any idea what the underlying issue could be? Could we help speed up the process? As I mentioned above, the error-messages are too non-descript for me to help you without further guidance 🙈

@martinRenou
Copy link
Contributor

This will be fixed by #643

@michaelweinold
Copy link
Contributor Author

Amazing, thank you!

@martinRenou
Copy link
Contributor

martinRenou commented Oct 13, 2023

You should be able to test it with pip install --pre jupyterlite jupyterlite-xeus-python

@michaelweinold
Copy link
Contributor Author

I tried to test this just now - works fine. Many thanks for the fix!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants