import numpy fails from script run with IPython %run magic

[pierre-haessig]

Hello,

I have a strange (i.e. only half reproducible) failure of import numpy in script run from a notebook with the %run -i IPython magic.

I have two deployments of jupyterlite/demo, with the same requirements.txt (pierre-notebooks/requirements.txt and Microgrids.web/requirements.txt) based on jupyterlite/demo#128 (15 Sept 2023).
In both repos, I have "no-code" notebook demos with interactive plots (matplotlib+ipywidgets) where most of the code is hidden in a setup script :

• https://pierre-haessig.github.io/pierre-notebooks/retro/notebooks/?path=Frequency%20regulation%20nocode.ipynb
• https://microgrids-x.github.io/Microgrids.web/retro/notebooks/?path=Microgrid_showcase.ipynb

The common structure of these notebooks is:

1. First cell installs ipywidgets + any other pure Python package. Not with %pip install but within a try: import piplite block so that the exact same notebook can also be run an classical (non-Lite) Jupyter Lab.
2. Second cell runs (with %run -i) a setup Python script which hides from the user import numpy as np and many lines of code.
3. Interesting part of the notebook

Now come the strange behavior:

• in the Microgrids.web / Microgrid_showcase notebook, running the setup script works
• in the pierre-notebooks / Frequency regulation notebook, running the setups fails to import numpy with the following error:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
File /drive/Frequency_regulation_setup.py:5
      1 # Setup code for the "Frequency regulation nocode.ipynb" notebook.
      2 ...
----> 5 import numpy as np

ModuleNotFoundError: The module 'numpy' is included in the Pyodide distribution, but it is not installed.
You can install it by calling:
  await micropip.install("numpy") in Python, or
  await pyodide.loadPackage("numpy") in JavaScript
See https://pyodide.org/en/stable/usage/loading-packages.html for more details.

Now, if I create a new cell with import numpy, this import works fine and then I can rerun the cell which runs the script (which then fails on the next imports: scipy and matplotlib...)

I understand from the Pyodide doc loading package mentioned in the error message that there is a special import hook in the REPL that automates the piplite.install call for pre-packaged packages. Perhaps then it is not recommented to import such packages in a %run -i run script?

The strange thing is that it works in one JupyterLite deployment and not in the other. I could only spot small differences between the two:

• in the Microgrids.web / Microgrid_showcase notebook:
  • the first cell await installs ipywidgets + another pure Python package from PyPI
  • the filename setup script has no white spaces
• in the pierre-notebooks / Frequency regulation notebook
  • the first cell await installs only ipywidgets
  • the filename setup script has white spaces (so that the filename in %run -i is surrounded by quotes)

→ this starts looking like the game https://en.wikipedia.org/wiki/Spot_the_difference... ;-)

Context

Versions:

• JupyterLite version: 0.1.2
• jupyterlite-pyodide-kernel 0.1.2
• Operating System and version: Ubuntu 20.04
• Browser and version: both Firefox and Chromium, after having flushed the browser storage

[bollwyvl]

Yes: the loadPackagesFromImports of pyodide standard distribution packages only works at the plain python AST level. This mechanism doesn't work with nested imports (e.g. import my_script which in turn import numpy), and %run is pretty much that... and can also be dynamically expanded with variables from get_ipython().user_ns.

This kernel exerts a bit more control over %pip, a line magic, and requirements files can be nested, so it could be like:

if sys.modules.get("pyodide"):
  %pip install -r requirements-numpy.txt 

Where the new requirements file was like:

numpy
-r requirements.txt

Which would load:

ipywidgets
# whatever else...

Another path is to build a wheel that encapsulates the needed dependencies, and ship that in PyodideAddon.piplite_urls so there is only one, well-defined import to bring all the dependencies, but at present it would still need to be installed with piplite.install or %pip. Such a package could even have [extra] fields to support e.g. %pip install mydemo[numpy].

[pierre-haessig]

OK. Thanks for the feedback, although to be honest, I got lost towards the middle!

Anyway, in the meantime, I've inserted a "collective" import numpy, scipy, matplotlib cell which "preheats the machinery" before running my code setup script.

I agree that at some point it would be nicer to prepackage all dependencies, to avoid my demo from breaking repeatedly! However, for the moment, I enjoy the "opaque simplicity" of the deployment based on Github actions (thanks to the demo template repo). I don't fully understand how it works, but it mostly works!

[martinRenou]

Another approach would be to use jupyterlite-xeus-python which allows you to pre-package any dependency that you need. You just need an environment.yml file specifying your dependencies.

[pierre-haessig]

Another approach would be to use jupyterlite-xeus-python which allows you to pre-package any dependency that you need. You just need an environment.yml file specifying your dependencies.

Yes, I saw this mentioned in other threads. However, one of the demo uses a pure Python package of mine which I only published on PyPI (while jupyterlite/xeus-python-kernel needs conda-forge). But in one of these threads I saw this limitation was to be overcome, so that would become very useful indeed!

[martinRenou]

Pip packages is now supported since recently (see this section of the docs) though it installs them with the --no-deps option due to some limitations so you'd need to specify your dependencies in the environment.yml file as well (better installed with conda/mamba than pip)