DOC Add JupyterLite button in example gallery

[lesteve]

This is a draft PR to use sphinx-gallery Jupyterlite integration sphinx-gallery/sphinx-gallery#977 and detect possible issues.

Unfortunately CircleCI does not allow JupyterLite to work in artifacts, so for now I will manually run make html-noplot locally and make the output accessible at http://lesteve.github.io/scikit-learn.

See sphinx-gallery/sphinx-gallery#977 (comment) for more details about the CircleCI limitation.

[thomasjpfan]

Is this PR blocked by sphinx-gallery/sphinx-gallery#1100 and resolving the tests failures in https://github.com/lesteve/scikit-learn-tests-pyodide?

I can see there is some ongoing work to get SciPy + OpenBLAS to work: pyodide/pyodide#3331 aiming to resolve some of the WASM tests failures in scikit-learn.

[lesteve]

Is this PR blocked by sphinx-gallery/sphinx-gallery#1100 and resolving the tests failures in lesteve/scikit-learn-tests-pyodide?

I would hope not 😉. In my opinion, there are a few ways towards this PR getting merged even without the Scipy and scikit-learn tests passing on Pyodide (this could take a while to get them to pass if I am being honest):

• manage user expectation e.g. by:
  • adding a cell with a big warning at the beginning of each notebook (only in JupyterLite of course) saying that this is beta and may break. We could also try to indicate this beta status already near the JupyterLite button, suggestions more than welcome!
  • black-listing some examples that we know don't work and find a way to indicate (maybe with a greyed out button) that it is not meant to work. The problem is that it is not so easy to know which examples are not meant to work and some failures are not deterministic e.g. see Fatal error with snippet using np.random and scipy.linalg pyodide/pyodide#3203. In principle you can run all the examples in a similar fashion I am running the scipy and scikit-learn test suite but this will probably take even longer than on CPython. This could be based on user feed-back too.
• the missing dependency example is something I have some ideas about how to deal with, for example by adding cells at the beginning of the notebook with %pip install, stay tuned for this! It may be enough for the scikit-learn doc although probably not enough for the generic case. Using the xeus-python kernel with emscripten-forge may be what is needed for the generic case.

I can see there is some ongoing work to get SciPy + OpenBLAS to work: pyodide/pyodide#3331 aiming to resolve some of the WASM tests failures in scikit-learn.

Yes this too will take a while. There is some hope that it may fix some of the weird issues we see, but it is hard to tell without doing the (hard and long) work ...

[lesteve]

To work around the CircleCI limitation, I made the output of a local run make html-noplot at http://lesteve.github.io/scikit-learn. I am planning to update that manually for now.

Here is one particular example, if you want to play with it: http://lesteve.github.io/scikit-learn/stable/auto_examples/release_highlights/plot_release_highlights_1_2_0.html

[thomasjpfan]

Thanks for the update. The "launch lite" button is giving a 404:

http://lesteve.github.io/scikit-learn/stable/auto_examples/lite/lab/?path=auto_examples/release_highlights/plot_release_highlights_1_2_0.ipynb

[lesteve]

Indeed the bug is fixed in sphinx-gallery development version, the right URL is http://lesteve.github.io/scikit-learn/stable/lite/lab/?path=auto_examples/release_highlights/plot_release_highlights_1_2_0.ipynb

[betatim]

Having a "beta" label on the button would be a good start I think (easier than blacklisting examples). If it leads to a huge number of complaints (instead of constructive issues) it is easy enough to remove the button again from the docs. So I think we should do what Loic suggested and not wait for all things to be resolved.

[thomasjpfan]

adding a cell with a big warning at the beginning of each notebook (only in JupyterLite of course) saying that this is beta and may break.

Once the dependency issue is resolved, I'm okay with the big warning option at the beginning of each notebook that only appears in JupyterLite.

[lesteve]

So right now I have added warnings at the top of each notebook like this:

There is also a cell for JupyterLite-specific code that should cover most of the use-cases:

• installing packages like seaborn or plotly with %pip install + some more code e.g. in the plotly case
• using pyodide_http.patch_all() to patch HTTP requests and make things like sklearn.datasets.fetch_* work

I put a red warning for the one notebook that uses Plotly because I think Plotly has issues in JupyterLite (although the notebook seems to be working for some reason)

I think the main thing that would be needed to get this merged is a sphinx-gallery release. At the same time it is good to stress-test sphinx-gallery a bit more before doing a release ...

What else do you think is needed to get this merged?

• Is a beta near the Jupyter logo a must? I could probably hack something that does not need a sphinx-gallery modification
• is the "open JupyterLite in a new tab" a must? Not sure whether it can be hacked without modifying sphinx-gallery (maybe with some Javascript that adds on the fly "target=_blank". Not sure I want to make sphinx-gallery more complicated with an additional option since it seems like a JupyterLite bug that may be fixed one day (although maybe not super soon).
• Tweak the wording of the warning and maybe add a link to the scikit-learn issues in case they noticed issues. I don't think this will generate a lot of traffic but I may be wrong
• Something else?

Reminder: you can look at an example here (CircleCI limitation prevents JupyterLite to work inside the CircleCI artifacts):
https://lesteve.github.io/scikit-learn/stable/auto_examples/release_highlights/plot_release_highlights_1_2_0.html#sphx-glr-auto-examples-release-highlights-plot-release-highlights-1-2-0-py

[adrinjalali]

I just tried the linked example, running the cells, it says pandas is missing from the environment, and !pip install pandas fails too.

[thomasjpfan]

Thank you for the update. This is looking good!

Is a beta near the Jupyter logo a must?

I think the banner in the notebook is enough.

open JupyterLite in a new tab

I think the current behavior is okay. The binder link does not open in a new tab either. (I'll prefer to open in another tab, but it's not a blocker.)

As for the example: lesteve.github.io/scikit-learn/stable/auto_examples/release_highlights/plot_release_highlights_1_2_0.html#sphx-glr-auto-examples-release-highlights-plot-release-highlights-1-2-0-py, I'm getting a BadStatusLine: HTTP/1.1 0 error running fetch_openml:

fetch_openml(
    "titanic", version=1, as_frame=True, return_X_y=True, parser="pandas"
)

[lesteve]

Thanks a lot for your feed-back:

• the pandas one should be not too hard to fix (add import pandas in the relevant notebooks). %pip install pandas is the way to go for JupyterLite and probably other notebook interfaces too.
• the fetch_openml one may be down to a CORS issue, I asked OpenML in CORS policy openml/OpenML#1135 (comment) if the right headers can be added to the server response.

[thomasjpfan]

Does JupyterLite require an "import pandas" before a library can use it?

In any case, I am okay with running "import pandas" all the time to prevent any pandas related issues.

[lesteve]

My understanding is that there is a bit of magic in Pyodide that if it you type import matplotlib in the Pyodide console (or in JupyterLite with a Pyodide kernel in our case) it will "just work". This is the case for all packages that are included inside Pyodide, including pandas.

On the other hand if you are calling a function that does an import inside it it will not work. Examples where this happens:

• from sklearn.utils import check_matplotlib_support; check_matplotlib_support('test')
• %matplotlib inline

A work-around is to do the import before calling the function.

The kind of error you get:

ModuleNotFoundError: The module 'matplotlib' is included in the Pyodide distribution, but it is not installed.
You can install it by calling:
  await micropip.install("matplotlib") in Python, or
  await pyodide.loadPackage("matplotlib") in JavaScript
See https://pyodide.org/en/stable/usage/loading-packages.html for more details.

I am not sure how much hope there is too fix it in Pyodide. I would agree that this is indeed suprising.

[thomasjpfan]

Wow, I see. In that case, can we work around these issues by always importing packages that is commonly use in the examples, such as matplotlib, pandas, etc.?

[lesteve]

I guess that would be another option indeed, I was trying to add some code only in the notebooks where it was needed but maybe this is not worth the effort ...

Also I opened pyodide/pyodide#3771 to get some feeling whether the underlying issue is fixable.

[thomasjpfan]

With #26171 merged, is this patch required?

[lesteve]

Yes for now until the OpenML header thing is fixed.

Note that the scikit-learn version we are using is the version available in Pyodide, namely 1.2.2 at the time of writing and #26171 only is in main.

I have some medium-term plan to add a CI build for the scikit-learn development version, which potentially could be published similarly to our scikit-learn development wheel and installed in JupyterLite but it will take a bit of time.

[thomasjpfan]

To check my understanding: if there was a Pyodide build that includes #26171, can this OpenML work-around be removed without waiting for openml/OpenML#1135 to be resolved?

[lesteve]

Yes, although my favourite way out would be that the OpenML CORS issue can be fixed in the not too distant future and this temporary work-around can be removed.

[lesteve]

I think this is in a good enough state to be reviewed, I have updated https://lesteve.github.io/scikit-learn, the 1.2 release highlight example runs fine.

Feel free to run other examples from the gallery and find other issues 😉!

[adrinjalali]

I tried a few examples, they all ran nicely, except that for some of them they'd get stuck in a cell (not necessarily the import one), and after a kernel restart, they run fine. This is NICE!

[lesteve]

I tried a few examples, they all ran nicely, except that for some of them they'd get stuck in a cell (not necessarily the import one), and after a kernel restart, they run fine.

Interesting, if you remember the particular notebooks that get stuck, I can have a closer look. Now that using OpenBLAS pyodide/pyodide#3331 has been merged, the full scikit-learn pass (except a few xfailed tests) with the Pyodide development version see https://github.com/lesteve/scikit-learn-tests-pyodide for more details, so I am hoping this kind of intermittent issues will be a thing of the past.

This is NICE!

Thanks 😉!

[adrinjalali]

if you remember the particular notebooks that get stuck, I can have a closer look

It's not something I can reproduce, so I'm happy to have this merged and have a broader testing audience and take it from there.

[adrinjalali]

Otherwise LGTM.

[thomasjpfan]

I'm trying to see how this alters our release process. At a high level, I see this process:

1. Cut a release with scikit-learn v1.3
2. Update Pyodide to build scikit-learn v1.3 (hopefully with no additional patches)
3. Wait for Pyodide to release so that it includes scikit-learn v1.3

Is this correct? If so, my concern is the wait for Pyodide to release. There will be a time period where our examples were updated to v1.3, but not run on the version packaged by Pyodide.

[adrinjalali]

Is this correct? If so, my concern is the wait for Pyodide to release. There will be a time period where our examples were updated to v1.3, but not run on the version packaged by Pyodide.

Would there be a way for the examples to specify exact version of scikit-learn they want installed, and fail if it's not released, with a nice message that users need to wait for that, and in the meantime they can download the notebooks for themselves or use mybinder?

[lesteve]

Is this correct? If so, my concern is the wait for Pyodide to release. There will be a time period where our examples were updated to v1.3, but not run on the version packaged by Pyodide.

Good point indeed! I can envision a future where we build a Pyodide wheel in our CI and we upload it somewhere (TBD) and then we could add %pip install <the-right-incantation-goes-here> in the notebooks to make sure you install the right version on scikit-learn rather than relying on the one from Pyodide.

Full disclosure, I have not done it so I may be oversimplifying the situation ... it seems like pydantic-core has wasm wheels in its github release https://github.com/pydantic/pydantic-core/releases/tag/v0.25.0 but for now PyPI does not support wasm wheels e.g. https://discuss.python.org/t/support-wasm-wheels-on-pypi/21924.

cc @rth in case I am oversimplifying the situation too much on the Pyodide wheel packaging and installing part.

Before this future happens, the process you mention seems to describe accurately what would happen. There is an additional thing that I am not sure about, is how JupyterLite controls the Pyodide version. Right now it seems like JupyterLite has Pyodide 0.23.0 although Pyodide 0.23.1 has been released a few days ago.

Would there be a way for the examples to specify exact version of scikit-learn they want installed, and fail if it's not released, with a nice message that users need to wait for that, and in the meantime they can download the notebooks for themselves or use mybinder?

I guess we could add a cell that checks the version at the beginning with a nice error message if the scikit-learn version is not recent enough. You would still need to figure out (and maintain) which version is needed by each example but maybe this is worth it.

[adrinjalali]

I'm a bit confused, can we not have a specific version installed in that environment? Let's say two releases down the line, what happens if the user looks at an example from an older version, and clicks on the button, do they get an environment with the most recent version instead of the one corresponding to the example?

[lesteve]

I'm a bit confused, can we not have a specific version installed in that environment?

Right now, JupyterLite has a fixed Pyodide version, and Pyodide has a fixed version of scikit-learn (1.2.2 for Pyodide 0.23.0 for example). As I was trying to say we could build ourselves a scikit-learn wasm wheel, make it accessible somewhere so that it can be installed in the notebook via %pip install <the-right-thing-goes-here>.

Let's say two releases down the line, what happens if the user looks at an example from an older version, and clicks on the button, do they get an environment with the most recent version instead of the one corresponding to the example?

I am not too sure how that would work, but at first sight it seems non trivial to support the older version use case with JupyterLite. I would guess this use case is very rarely used. In an ideal world I agree we would support at the very least stable vs dev though.

Checking the Binder link behavior, you actually get a scikit-learn development version, even from the stable doc (and even from older versions). Nobody seems to have complained about it yet 😉.

[betatim]

Is it feasible to build pyodide (or JupyterLite? or both?) with the specific version of scikit-learn that the docs were built with? Then we could ship everything needed, new versions would work and old ones would keep working as well because everything is hosted in the build output of the docs page.

[lesteve]

I guess my preference would be for this PR to be merged in its current state and then to iterate in further PRs to make it more robust. I feel like merging this PR would be useful to gather wider feed-back and it is unlikely to be too disruptive since it is only on the dev website. I am quite hopeful that there is a way forward to make it more robust and that it will work by the time the next scikit-learn release comes around.

I would totally understand if cautiousness is preferred though and that the consensus is making it more robust before merging this PR.

Right now I think the best way forward looks like this:

• build a development wheel in our CI. This needs to use the right emscripten version to match the Pyodide one use below.
• upload it somewhere where it can be used inside JupyterLite via %pip install, I think that means github right now because you need a CORS proxy and cnd.jsdelivr.net/gh gives us one for github files.
• set the Pyodide version in JupyterLite to match the right emscripten version (there must be a way to do this although I still need to figure out how)

[thomasjpfan]

I think the big warning box is good enough to show that jupyterlite integration is experimental.

I'm happy with merging this now and iterate. LGTM

[lesteve]

Great thanks a lot!

[ogrisel]

This is a really great achievement. Thanks so much to anyone involved and @lesteve in particular!

[rth]

As I was trying to say we could build ourselves a scikit-learn wasm wheel, make it accessible somewhere so that it can be installed in the notebook via %pip install .

Yes, that should mostly work already now. The work needed is to integrate that setup nicely with sphinx-gallery and Jupyterlite, as that's not a typical workflow.