CI, BLD: Push NumPy's Emscripten/Pyodide wheels nightly to Anaconda.org PyPI index

[agriyakhetarpal]

Description

This PR adds a schedule to push WASM wheels that are compiled via the Emscripten toolchain and Pyodide ecosystem to NumPy's PyPI-like index on Anaconda.org at https://anaconda.org/scientific-python-nightly-wheels/numpy.

Key changes

1. The addition of a schedule to the job, which is the same as the schedule that wheels.yml runs on
2. A workflow_dispatch: trigger has been added to push the wheels manually if needed
   a. The input is false by default to not accidentally upload and subsequently overwrite any existing wheels.
3. A step has been added that runs after the tests run and succeed, which uses the NUMPY_NIGHTLY_UPLOAD_TOKEN repository secret. These uploads will not be attempted on forks or on workflow run contexts outside of the provided condition(s) in this newly added step.

[agriyakhetarpal]

For more context to offer to reviewers, here: @rgommers suggested using the scientific-python/upload-nightly-action GitHub Action and to keep the workflow and the upload in a single place, to establish consistency with other repositories that are uploading (or will be uploading) these WASM wheels. This should be ready for review as-is, and it does not need to access NUMPY_STAGING_UPLOAD_TOKEN or other repository secrets, since the multibuild-wheels-staging organisation is not being uploaded to at this time.

[lesteve]

Just curious, is there a way to actually conveniently use the uploaded wheels e.g. through %pip install inside JupyterLite (or micropip.install inside Pyodide console)?

It seems like there is still the issue that CORS headers are still to be added to anaconda.org. I haven't tried in a while but this one week old comment pyodide/micropip#101 (comment) seems to say this is still the case. Related to pyodide/pyodide#3049 (comment).

[agriyakhetarpal]

It seems like there is still the issue that CORS headers are still to be added to anaconda.org. I haven't tried in a while but this one week old comment pyodide/micropip#101 (comment) seems to say this is still the case. Related to pyodide/pyodide#3049 (comment).

Hi @lesteve – I have been in the works for adding these nightly wheels to PyWavelets (see PyWavelets/pywt#728) where I put up the linked request. But since there has been a response there from the Anaconda side, maybe the third time's the charm?

For alternatives, we had previously considered using the jupyterlite-xeus kernel instead of jupyterlite-pyodide-kernel because the former has support for pre-installing packages before the documentation is built, but that right now requires setting up and maintaining a conda recipe where artifacts will then be downloaded off of emscripten-forge. Doing that does not sound viable at this time, so maybe xeus-kernel can include support for Python wheels sometime soon.

Just curious, is there a way to actually conveniently use the uploaded wheels e.g. through %pip install inside JupyterLite (or micropip.install inside Pyodide console)?

However, to answer this – you should—in theory—be able to use these nightly wheels by downloading them from GitHub Actions artifacts and letting micropip handle it from a local server?

[agriyakhetarpal]

Converting this PR to a draft until there is a fix for #26164

[agriyakhetarpal]

Since there is a fix now, rebasing and marking the PR as ready for review again.

[lesteve]

Context : I was interested as well to have an easily installable Pyodide wheel for the scikit-learn documentation, one for the latest scikit-learn release (independently on Pyodide release cycles) one for the scikit-learn development version.

It would be nice if anaconda.org had CORS headers set-up, but the fact that it has not happened yet, may indicate that there are some complications. One of the nice thing with anaconda.org is that they have pypi.anaconda.org so that would allow to use the Pyodide wheel conveniently like micropip.install('scikit-learn', index_urls=['https://pypi.anaconda.org/...')

As for the alternatives I have personally considered but left for later for now:

• it seems like the Pyodide kernel can also ship Pyodide wheels at build-time https://jupyterlite.readthedocs.io/en/latest/howto/pyodide/wheels.html. The wheel would be part of the doc website, which adds a few MB but maybe OK enough
• have the Pyodide wheel in another github repo so you can do things like this in a notebook. The URL is quite long which is not ideal:

  %pip install https://cdn.jsdelivr.net/gh/lesteve/scikit-learn-tests-pyodide@scikit-learn-wheel/dist/scikit_learn-1.2.dev0-cp310-cp310-emscripten_3_1_21_wasm32.whl
  

• @rth did a quick-and-dirty CORS proxy with Cloudfare at one point. I guess if someone (maybe from Scientific Python?) wanted to maintain such a CORS proxy this could be an option?

[agriyakhetarpal]

it seems like the Pyodide kernel can also ship Pyodide wheels at build-time jupyterlite.readthedocs.io/en/latest/howto/pyodide/wheels.html. The wheel would be part of the doc website, which adds a few MB but maybe OK enough

We were thinking of doing this, too – however, we did not get to a point where we could reliably download the wheel to be shipped at build time (most likely one would have to do that by overriding the build commands of the Read the Docs build inside the readthedocs.yaml file). I imagine that doing so on a docs site that uses GitHub Pages would be a bit easier, though.

have the Pyodide wheel in another github repo so you can do things like this in a notebook. The URL is quite long which is not ideal:

Would this be easier to track with a GitHub Release? GitHub sets CORS headers, so tools like pip and pipx are generally able to download and install wheels from URLs like this, and a release URL would be a bit shorter if that helps. I do not have much experience with web dev, so I am not sure how setting up a CORS proxy goes :)

In the best case, yes, having a wheel come pre-installed/pre-loaded with jupyterlite-pyodide-kernel should solve those problems, but I cannot comment on how easy that would be, or where one would start with the implementation. If the Xeus kernel can do it, then I suppose the Pyodide kernel can, too.

[charris]

Thanks @agriyakhetarpal .