[ENH]: out-of-tree Pyodide builds in CI for Matplotlib #27870
Comments
|
I already have a branch on my fork of Matplotlib where a few workflow runs have succeeded in the compilation and the wheel is being built: https://github.com/agriyakhetarpal/matplotlib/actions/runs/8161780336/job/22311265339. I am working on running the tests, which are failing because of circular imports. It seems to be picking up the current (root) directory – I would appreciate any suggestions on how to run the test suite in this case (it works locally, but we are limited by the fact that the |
|
Based on https://matplotlib.org/devdocs/devel/testing.html#testing-released-versions-of-matplotlib, I am assuming I need to additionally check out the GitHub repository and copy the baseline images to where I installed the Matplotlib wheel... Edit: done in https://github.com/agriyakhetarpal/matplotlib/actions/runs/8173325260/job/22345664858 where I am downloading the built wheel artifact in another job (that needs the first one to pass) and doing a sparse checkout of the baseline images to copy to the Matplotlib installation inside the virtual environment. I am facing issues with test collection – |
|
Another progress update via this workflow run – tests are now being collected, but fail to run because Matplotlib fails to be imported – this is due to the existence of the It is to note that the in-tree Pyodide build does contain patches to repeal the use of threads, which I can adopt:
Footnotes |
|
Some more progress: I added a patch to disable the use of threading and skipped certain tests that were importing the threading standard module (threading is not supported in Pyodide yet). Next, I have split the build and test steps into two separate jobs that share artifacts (the wheel, in this case). The testing job then checks out the baseline images from the Matplotlib repository and copies them into the installed Matplotlib inside the virtual environment set up by Pyodide, and runs the It seems that my patch is not entirely correct, because the tests seem to proceed to collect, but fail and the interpreter prints out a random seed (supposedly set by NumPy) without any other stack trace. This makes it difficult to debug what is happening. Here are the logs: https://github.com/agriyakhetarpal/matplotlib/actions/runs/8187758884/job/22390718867#step:11:43 |
|
We used to have the threading imports protected, but removed them in #23073 because we thought pyodide no loger needed them. Happy to take those back in if needed. |
|
Hi @tacaswell, I did have further progress after my last comment and disabled the threading imports where needed – but it looks like this could be, and remain, stalled for a while due to some unresolved symbols (please see the linked issue above) where my inexperience beats me. I do have working patches and a reliable CI job in a branch on my fork where the rest of the build procedure seems to be working, though. It isn't ready for a PR yet until I have a fully working test suite that completes itself, but I'm more than happy to open a draft PR if by doing so it can meet more eyes (and possibly drive progress towards resolution). |
|
I'd switch to the testing method that scikit-learn uses to see if that fixes the missing symbol issues. If that doesn't help, let's look at them in more detail. |
|
@thomasjpfan and I were looking at this at the Dev Summit yesterday. Unfortunately, I forgot that this issue existed, and somewhat repeated what you've already done, but fortunately haven't done much that would replace it. I do think we can take on a couple of the patches more generally instead of leaving them downstream. One thing that has made this a bit easier is that cibuildwheel just merged support for building wasm wheels. There's been some hacks, but I'm able to reach a point that I can build locally and import, but plotting fails with a crash in FreeType trampolining. Here is the backtrace of that event: "matplotlib/backends/backend_agg.py", line 220 in If you happen to know how to get wasm to compile with a bit more debug information, that could maybe help me move forward a bit. |
|
Thank you, @QuLogic and @thomasjpfan! I haven't looked at this in a while but this remains very much on my radar. @ryanking13 had previously shared https://pyodide.org/en/stable/development/debugging.html with me which has some information on function signature mismatches, where compiling with |
|
Thanks, I've looked through that page and managed to find some info about the wasm debug tools that was helpful. We need to disable some of our build options in order to get more symbols (more than just passing |
Problem
Hi there! I am opening this feature request to gauge ideas and comments about out-of-tree Pyodide builds, i.e., wasm32 wheels via the Emscripten toolchain for Matplotlib. In my most recent work assignment, I am working on improving the interoperability for the Scientific Python ecosystem of packages with Pyodide and with each other, which shall culminate with efforts towards bringing interactive documentation for these packages where they can then be run in JupyterLite notebooks, through nightly builds and wheels for these packages pushed to PyPI-like indices on Anaconda, at and during a later phase during the project.
This project is being tracked at Quansight-Labs/czi-scientific-python-mgmt#18.
Proposed solution
This issue proposes out-of-tree builds for
matplotlibon its own CI and build infrastructure. I would be glad to work on this. This is how it would proceed, tentatively:matplotlibare pursuedThe text was updated successfully, but these errors were encountered: