Rebuild netcdf4; Rebuild libnetcdf; fix runtime failure due to missing listxattr#5951
Open
MMesch wants to merge 4 commits into
Open
Rebuild netcdf4; Rebuild libnetcdf; fix runtime failure due to missing listxattr#5951MMesch wants to merge 4 commits into
MMesch wants to merge 4 commits into
Conversation
MMesch
force-pushed
the
fix-netcdf4-dimscales
branch
2 times, most recently
from
July 20, 2026 15:17
f4548c4 to
5495139
Compare
Contributor
Author
|
@IsabelParedes , waiting for your thoughts on this now. After discussing with @DerThorsten these are some points to consider:
|
Member
This would be nice. Thanks! |
- bump 1.12.3 -> 1.14.6, update source URL template - patches/0001-fenv-fe-invalid-guard.patch: guard FE_INVALID use in H5Tinit_float.c; emscripten's <bits/fenv.h> defines only rounding modes, FE_INVALID is not declared. This patch is already upstream in hdf5 2.x and can probably be dropped once we update further https://github.com/HDFGroup/hdf5/blob/f8844a64c639f7ba33592a7948d4adacacb2d451/src/H5Tinit_float.c#L614-L617 - patches/0002-drop-soversion.patch: skip VERSION/SOVERSION properties under EMSCRIPTEN so libhdf5_hl.so records bare "libhdf5.so" as its NEEDED entry. Otherwise it records "libhdf5.so.310.5.1" while other packages link with -lhdf5 and record "libhdf5.so". The emscripten loader seems to dedupe by string identity (not realpath) and ends up loading libhdf5.so twice. One visible symptom of this was that the hdf5 metadata cache errors when calls cross between those two loaded instances, something which blocks simple file reading and also affects netcdf4 transitively. Note that this is a hot fix and a proper solution that also works for other libraries would be nicer to have. - variant.yaml: pin hdf5 to 1.14.6 - 1.14 auto-generates settings/ so the old workaround is dropped
MMesch
force-pushed
the
fix-netcdf4-dimscales
branch
from
July 23, 2026 13:55
5495139 to
82c366e
Compare
- HDF5_VERSION 1.12.3 -> 1.14.6 - drop the ctypes.CDLL preload prepended to __init__.py; the hdf5 SOVERSION fix makes the emscripten loader dedupe libhdf5.so properly, so the manual dlopen seems no longer needed. - add version check and a dimension-scale round-trip test that guards the HL path (H5DSset_scale + attach_scale + is_scale after reopen)
netcdf-c's isdaoscontainer() in libdispatch/dinfermodel.c uses listxattr/getxattr to sniff a path for DAOS marker attributes. Emscripten's <sys/xattr.h> declares these but the runtime does not implement them, so the built .so fails to load with "cannot resolve symbol listxattr". There is no --disable-daos configure option (the check is hard-coded), so patch the source to early-return NC_ENOTNC under __EMSCRIPTEN__. Also bump the variant.yaml pin to 4.10.1 to match the recipe version; the previous 4.9.3 pin was forcing consumers to resolve against upstream 4.9.3 which had been built against HDF5 1.12.3 headers and aborted at runtime against a 1.14.6 runtime.
netcdf-4 file access walks the HDF5 tree with H5DSis_scale to distinguish coordinate dims from data variables. Without -lhdf5_hl in the extension LDFLAGS the built _netCDF4.so has no libhdf5_hl.so in its NEEDED entries, so the loader never brings it in and calls to H5DSis_scale die with "Dynamic linking error: cannot resolve symbol H5DSis_scale" the moment xarray/netCDF4 opens a NETCDF4 file. Also replace the smoke test with two focused cases: a NETCDF3_CLASSIC round-trip and a NETCDF4 round-trip that exercises dim + variable + attribute together.
MMesch
force-pushed
the
fix-netcdf4-dimscales
branch
from
July 23, 2026 14:40
82c366e to
f682d4f
Compare
3 tasks
MMesch
changed the title
Contributor
@MMesch isnt that the culprint / root-problem @MMesch did you trace the stuff in the browser to see that |
MMesch
changed the title
Contributor
Author
|
@DerThorsten , I split out hdf5 to PR #6007. I'll write a more detailed description what I did in there. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Edited after splitting out new PR #6007 .
Summary
Fixes the runtime failure
xarray.open_dataset(...)→netCDF4.Dataset→nc_open→H5DSis_scale→Dynamic linking error: cannot resolve symbol H5DSis_scalein the browser, and the follow-onRuntimeError: ring type mismatch occurred for cache entrythat surfaces onceH5DSis_scaleis reachable.Build Notes
-lhdf5_hlto LDFLAGS. This may not strictly be necessary since it seems to be activated by default. But it's probably safer to add. The bug we observed was related to this library.__EMSCRIPTEN__: netcdf-c 4.10.1 referenceslistxattrfromlibdispatch/dinfermodel.cinside the DAOS-container probe, guarded byHAVE_SYS_XATTR_H. Emscripten's<sys/xattr.h>declareslistxattrbut the runtime doesn't implement it, so the built.sofails to load.configure.acseems to have no DAOS knobs. We therefore patch the source to early-returnNC_ENOTNCunder__EMSCRIPTEN__. Attempting to set the variableac_cv_header_sys_xattr_h=noproduced garbled compile errors in the surroundingisdaoscontainer()function under-std=gnu23and was abandoned.variant.yamlbumped:hdf5: 1.14.6,libnetcdf: 4.10.1.