CI: WASM Pyodide#1903
Merged
Merged
Conversation
ax3l
force-pushed
the
wasm-pyodide-ci-tests
branch
2 times, most recently
from
5ca8125 to
5d1a7a0
Compare
Add a WASM CI job that cross-compiles the openPMD-api Python wheel for Pyodide (openPMD_USE_PYTHON=ON) and runs the full Python unittest suite (APITest, incl. the HDF5 read/write path) in the Pyodide test runtime via cibuildwheel's own test runner -- so the test ABI matches the freshly built wheel. Catches wasm-specific regressions a wheel smoke test cannot (e.g. the HDF5 64-bit long double read path, openPMD#1902). .github/ci/wasm_deps.sh cross-builds static zlib + HDF5 into the Emscripten sysroot (Emscripten restricts find_package to the sysroot) incl. the HDF5 FE_INVALID emscripten patch. run_python_tests.py sets cwd + sys.path so the suite's relative sample paths and API.APITest import resolve. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ax3l
force-pushed
the
wasm-pyodide-ci-tests
branch
from
5d1a7a0 to
6e756f7
Compare
ax3l
commented
Jun 29, 2026
Reproduce the ImpactX WASM teardown fault in openPMD-api's own CI: h5py bundles
a second static HDF5, which under Pyodide's single namespace interposes with
openpmd_api's HDF5. At interpreter teardown the tangled type registry makes the
openPMD HDF5 handler-destructor's H5Tclose fault ("not a datatype" -> wasm OOB),
even though the round-trip succeeds.
The probe runs after the full Python suite and is guarded with `|| echo` so a
crash is logged without failing the job while we reproduce (step 1). The guard
is dropped once the openPMD-api teardown fix lands, turning it into an enforced
regression test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
for more information, see https://pre-commit.ci
Run coload_repro.py first and standalone -- openpmd_api imported and run before h5py loads (the ImpactX order, openPMD as the primary HDF5) -- printing a marker per phase and both bundled HDF5 versions. This localizes any fatal fault (mid-operation vs. teardown) and distinguishes an ABI mismatch (h5py bundling a different HDF5) from the same-version teardown fault. Pin the import order with `# isort: skip_file` so pre-commit's isort does not alphabetize h5py ahead of openpmd_api (which would invert primary/secondary). Probe and suite are both guarded (`|| echo`, `;`) so both run and the job stays green while we diagnose from the log. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pyodide links the Python extension as a side module with -sSIDE_MODULE=1, which
whole-archives and force-exports every symbol -- including the statically linked
HDF5. A second co-loaded wheel that also bundles HDF5 (ImpactX, h5py, a second
openPMD) then cross-binds its HDF5 symbols into ours, so two HDF5 instances share
one tangled registry and crash on the first file op or at teardown ("not a
datatype" -> wasm memory access out of bounds). -fvisibility=hidden cannot
prevent this because the whole-archive export overrides it.
Mirror the APPLE `-exported_symbol` / Linux `--exclude-libs` co-load fix on
Emscripten: -sSIDE_MODULE=2 exports only the symbols in EXPORTED_FUNCTIONS (just
the module init _PyInit_openpmd_api_cxx), keeping HDF5 private to this wheel.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ty deps
The co-load crash (two wheels each embedding HDF5 in Pyodide's single namespace)
needs BOTH halves, neither of which suffices alone:
- -sSIDE_MODULE=2 on the extension: Pyodide's default -sSIDE_MODULE=1 whole-
archives and force-exports every symbol (so -fvisibility=hidden alone cannot
hide HDF5); =2 exports only _PyInit_openpmd_api_cxx.
- -fvisibility=hidden on the static zlib/HDF5 (wasm_deps.sh): makes their
definitions DSO-local so wasm-ld can relax our GOT references to HDF5 into
direct calls into our own copy, instead of binding to a co-loaded second
HDF5 (h5py, ImpactX) at load time.
Without the visibility half, =2 stops the re-export but our HDF5 calls still bind
to the first-loaded global HDF5 via the GOT and crash on the first file op; this
was verified in CI (the co-load probe still OOB'd with =2 alone).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Now that the co-load fix works, drop the `|| echo` guards so coload_repro.py and
the full suite are enforced regression tests: any wasm co-load regression fails
the job.
Also fix testScalarHdf5Fields, which ran for the first time on wasm (h5py is now
present in the test env): on wasm32 h5py writes the scalar with its native int
(int32) while np.array([45]) defaults to int64, so the store type-mismatched the
component ("store from Python array of type 'LONG' into Record Component of type
'INT'"). Store the component's own dtype instead of assuming the two agree.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ax3l
added a commit
to ax3l/openPMD-api
that referenced
this pull request
Jul 2, 2026
…E=2) library_builders.sh already builds the wasm zlib/HDF5 with -fvisibility=hidden (the DSO-local half). Add the complementary link-time half as a source patch: python-wasm-side-module.patch adds an EMSCRIPTEN branch to the co-load block (from python-hide-symbols.patch) that links the extension with -sSIDE_MODULE=2 and -sEXPORTED_FUNCTIONS=_PyInit_openpmd_api_cxx, so Pyodide no longer whole- archives and re-exports the bundled HDF5. Neither half fixes the co-load alone. Ports the dev fix openPMD#1903 (validated green there: the co-load probe + full Python suite run with a second HDF5, h5py, co-loaded). Applied in COMMON_PATCHES right after python-hide-symbols.patch, whose block it extends. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wrap the over-79-character comment lines flagged by the `style` job: coload_repro.py header, run_python_tests.py header, and the testScalarHdf5Fields dtype comment in APITest.py. No code changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- test/python/coload_repro.py (was .github/ci/): the co-load probe now lives with the other Python tests. - .github/workflows/dependencies/install_wasm.sh (was .github/ci/wasm_deps.sh): next to where dependency installers belong. - Drop run_python_tests.py: it only replicated cwd + sys.path[0]. Instead run `cd test/python/unittest && python Test.py -v`, mirroring the native ctest (CMakeLists.txt runs Test.py -v from that WORKING_DIRECTORY). Not pyodide- specific -- the ../samples relative paths require that cwd on every platform. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ax3l
commented
Jul 2, 2026
ax3l
commented
Jul 2, 2026
ax3l
added a commit
to ax3l/openPMD-api
that referenced
this pull request
Jul 2, 2026
…E=2) library_builders.sh already builds the wasm zlib/HDF5 with -fvisibility=hidden (the DSO-local half). Add the complementary link-time half as a source patch: python-wasm-side-module.patch adds an EMSCRIPTEN branch to the co-load block (from python-hide-symbols.patch) that links the extension with -sSIDE_MODULE=2 and -sEXPORTED_FUNCTIONS=_PyInit_openpmd_api_cxx, so Pyodide no longer whole- archives and re-exports the bundled HDF5. Neither half fixes the co-load alone. Ports the dev fix openPMD#1903 (validated green there: the co-load probe + full Python suite run with a second HDF5, h5py, co-loaded). Applied in COMMON_PATCHES right after python-hide-symbols.patch, whose block it extends. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ax3l
added a commit
to ax3l/openPMD-api
that referenced
this pull request
Jul 2, 2026
…d patch - setup.py version 0.17.1.post3 -> 0.17.1.post4 (rename the setup-py-version patch to match; update COMMON_PATCHES). - Regenerate python-wasm-side-module.patch so the applied CMakeLists co-load block matches openPMD-api#1903's current version (refined comment wording, no file-path reference, bare EMSCRIPTEN branch). Verified it applies cleanly after python-hide-symbols.patch and reproduces the openPMD#1903 block byte-for-byte. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ax3l
added a commit
that referenced
this pull request
Jul 2, 2026
* wheels: Windows std::mutex ABI fix (_DISABLE_CONSTEXPR_MUTEX_CONSTRUCTOR) The win64 openpmd_api wheel access-violates at `import openpmd_api` (NULL read in MSVCP140!Mtx_destroy) -- the VS 2022 17.10 std::mutex constexpr-constructor ABI break. The wheel --excludes msvcp*.dll, so the SYSTEM msvcp140.dll runs and can predate the build toolset's STL, which then NULL-derefs in Mtx_destroy. Surfaced co-loaded under the ImpactX wheel's win64 test: with the same fix added to the impactx build, amrex and impactx now import cleanly and the crash moved to the openpmd_api dependency -- which needs the identical fix. Add /D_DISABLE_CONSTEXPR_MUTEX_CONSTRUCTOR to the Windows CXXFLAGS to revert to the non-constexpr constructor, ABI-compatible with any msvcp140. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * win: set /D_DISABLE_CONSTEXPR_MUTEX_CONSTRUCTOR in library_builders.bat too The std::mutex ABI define is already in CIBW_ENVIRONMENT_WINDOWS (the central wheel build); set it in the Windows dependency builder as well so ADIOS2 (and any other C++ dependency) is built with the same non-constexpr std::mutex ABI as openPMD-api. Since we --exclude msvcp*.dll from the wheel, this keeps the whole toolchain consistent and portable against any system msvcp140.dll (VS 2022 17.10 made the ctor constexpr -> otherwise a NULL deref in Mtx_destroy). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * WASM: complete the co-load fix on the wheels branch (add -sSIDE_MODULE=2) library_builders.sh already builds the wasm zlib/HDF5 with -fvisibility=hidden (the DSO-local half). Add the complementary link-time half as a source patch: python-wasm-side-module.patch adds an EMSCRIPTEN branch to the co-load block (from python-hide-symbols.patch) that links the extension with -sSIDE_MODULE=2 and -sEXPORTED_FUNCTIONS=_PyInit_openpmd_api_cxx, so Pyodide no longer whole- archives and re-exports the bundled HDF5. Neither half fixes the co-load alone. Ports the dev fix #1903 (validated green there: the co-load probe + full Python suite run with a second HDF5, h5py, co-loaded). Applied in COMMON_PATCHES right after python-hide-symbols.patch, whose block it extends. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Wheels post4: bump version to 0.17.1.post4 + sync #1903 co-load patch - setup.py version 0.17.1.post3 -> 0.17.1.post4 (rename the setup-py-version patch to match; update COMMON_PATCHES). - Regenerate python-wasm-side-module.patch so the applied CMakeLists co-load block matches openPMD-api#1903's current version (refined comment wording, no file-path reference, bare EMSCRIPTEN branch). Verified it applies cleanly after python-hide-symbols.patch and reproduces the #1903 block byte-for-byte. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * simplify inline comment Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Wheels/RTD: use a supported python so the RTD skip actually cancels The .readthedocs.yml exit-183 skip never ran: build.tools.python "3.14" is not a supported Read the Docs build tool, so config validation failed and the per-PR RTD build errored (red check) instead of cancelling. post_checkout (exit 183) runs before the Python environment is created, so any valid version works -- use 3.12 so the config validates and the build cancels cleanly on the wheels branch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Wheels/RTD: make the exit-183 skip actually parse (add a build definition) The previous .readthedocs.yml (build.jobs.post_checkout: exit 183, with no sphinx/mkdocs/build.commands) was rejected at config parse time: the build failed right after `cat .readthedocs.yml`, before post_checkout ran, so exit 183 never fired and RTD posted a red check on every wheels-branch PR. The python version was a red herring -- 3.14 and 3.12 failed identically. Base the skip on dev's working config (add sphinx.configuration, use dev's os/python) so it validates; post_checkout's exit 183 then cancels the build before sphinx is read, and RTD reports the PR check as success (a cancelled build is a green check, per RTD docs). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI test for WASM Pyodide.
Ensure co-loading with
h5py, even when both use different HDF5 versions, works.