Comprehensive fixes for fill / sentinel-value handling

[maxrjones]

Fill-sentinel-values roadmap

Date: 2026-05-11 · Scope: 15 fill-sentinel-values issues + companion HDFParser audit.

TL;DR

• Correctness target = Zarr spec, not xarray. New test_zarr_spec_compliance.py is the primary check.
• Many "bugs" are upstream — xarray's FillValueCoder uses HDF5 base64 on the zarr backend. Tracked at xarray#11332. No PRs needed here for those.
• This week: merge #988, file companion-Virtual concatenation of arrays with different codecs or dtypes #5 issue, ship the small Phase-1 PR.

Phases

flowchart TD
    P1["<b>Phase 1</b> — parser fixes<br/><sub>stop the crashes · in progress (#988)</sub>"]
    P2["<b>Phase 2</b> — xarray upstream<br/><sub>FillValueCoder · no PRs here</sub>"]
    P3["<b>Phase 3</b> — h5py defaults<br/><sub>distinguish explicit fills</sub>"]
    P4["<b>Phase 4</b> — cross-parser<br/><sub>port HDF conventions</sub>"]
    P5["<b>Phase 5</b> — writer<br/><sub>Icechunk, Kerchunk, V2</sub>"]
    P6["<b>Phase 6</b> — attribute fidelity<br/><sub>JSON policy</sub>"]
    Meta["<b>#68</b> closed by totality<br/><sub>P1 + P3 + P4 + P5 + P6</sub>"]

    P1 --> P3 --> P4
    P1 -.-> P4
    P4 --> Meta
    P5 --> Meta
    P6 --> Meta

    P2 -.parallel, auto-resolves.-> Meta

    classDef done fill:#FAC775,stroke:#854F0B,color:#412402
    classDef upstream fill:#9FE1CB,stroke:#0F6E56,color:#04342C
    classDef todo fill:#D3D1C7,stroke:#5F5E5A,color:#2C2C2A
    classDef meta fill:#CECBF6,stroke:#534AB7,color:#26215C
    class P1 done
    class P2 upstream
    class P3,P4,P5,P6 todo
    class Meta meta

Loading

Next actions

1. This week
   • Merge #988.
   • File new issue: compound-dtype TypeError at hdf.py:364 (companion Virtual concatenation of arrays with different codecs or dtypes #5).
   • PR: compound-dtype guard + bytes→str decode + S-dtype _FillValue base64 (companion Set up CI #1).
2. In parallel (no PRs): track xarray#11332; add reproducers as variants surface. Comment on zarr-specs#351, zarr-extensions#33.
3. Then: Phase 3 (fill_value_defined() check).
4. Parallel after that: Phase 4 docs + Kerchunk hypothesis suite, Phase 5 writer round-trips.

Issues → cluster → phase

#	Title	Cluster	Phase
#22	Padding arrays with NaNs	F	5
#68	How to handle encoding	H	meta
#287	ZArray vs from_kerchunk_refs() disagreement	E	4
#339	NaN to parquet via to_kerchunk()	F	5
#352	Datetime NaT discrepancy	E	4
#478	V2 null fill_value coerced	F	5
#485	Example notebook FillValue decode error	C	2 (upstream)
#628	Icechunk _FillValue open failure	B/C	1 + 2
#715	Non-JSON-serializable attributes	G	6
#785	NetCDF _FillValue encoding error	B	1
#811	ZarrParser uint8 KeyError	A	1
#878	HDF5 dtype='<U1' failure	A + C	1 + 2
#904	DMRParser time parsing	E	4
#982	NetCDF3 _FillValue dropped	E	4
#989	fill_value: null decode	D	3 + 2 (upstream)

Clusters: A crash on extract · B HDF encoding gaps · C xarray upstream · D h5py default propagated · E cross-parser inconsistency · F writer-side · G attribute fidelity · H encoding meta.

Cluster details

A. Parser crashes during extraction — local, <20 lines each. HDFParser.fillvalue.item() on bytes (#878 — partial fix in #988), on None for vlen arrays (companion #6), structured dtype crash (companion #5). ZarrParser.get_metadata KeyError for uint8 (#811).

B. _FillValue encoding gaps in HDFParser — _construct_manifest_array skips encode_cf_fill_value for kinds S/U/O/T (hdf.py:98), but docs require base64 for S. Hits #785, #628.

C. xarray FillValueCoder — UPSTREAM. decode has no StringDType/|S*/O branch (→#485, #878), expects base64 for floats (→#989) and S bytes. Spec-compliance test passes; equivalence test fails as BothEnginesFailedIdenticallyError — that's the upstream attribution.

D. h5py default fillvalue propagated indiscriminately — _get_fill_value returns dataset.fillvalue unconditionally; h5py-default "" / 0 lands in zarr metadata. Cascades into vlen-string "corrupt buffer" (companion #2) and #989. Fix: dataset.id.get_create_plist().fill_value_defined().

E. Cross-parser inconsistency — HDF/Zarr/DMR/Kerchunk/NetCDF3 disagree on defaults and metadata for the same source (#287, #352, #982, #904). Architectural — needs a uniform plumbing contract.

F. Writer-side fill semantics — to_kerchunk() NaN in parquet (#339), V2 null fill_value coerced (#478), array padding feature request (#22).

G. Attribute serialization fidelity — non-JSON values break writers (#715), scalar dtype widens across JSON round-trip (companion #4).

H. Encoding model — #68, resolved by totality of A/B/D/E/F/G work.

Phase 1 — Local parser fixes

Task	Issue	Where	Size
Merge #988 (_get_fill_value + StringDType)	#878 mode 2	parsers/hdf/hdf.py	done
Compound-dtype guard	companion #5 (file issue)	hdf.py:364	5 lines
ZarrParser default-fill lookup	#811	parsers/zarr/...	5–15 lines
S-dtype _FillValue base64	#785, #628, companion #1	hdf.py:98-100	~10 lines

Validation: TestBasicEquivalence compound-dtype + S-dtype cases turn green. TestCompoundDtype fully green.

Phase 2 — Upstream advocacy (no PRs here)

test_zarr_spec_compliance.py already passes for these cases — we're correct, xarray needs to catch up.

Activity	Issues	Venue
Track FillValueCoder.decode JSON-native scalars	#485, #628, #878, #989, companion #1, #2	xarray#11332
Scalar→JSON encoding discussion	(framing)	zarr-specs#351
Optional[T] dtype	(long-term)	zarr-extensions#33
Document the gap	#989	docs/custom_parsers.md

Completes when: xarray ships. BothEnginesFailedIdenticallyError cases turn green automatically.

Phase 3 — h5py default distinction

Distinguish explicit vs h5py-default at _get_fill_value (hdf.py:38-51) via dataset.id.get_create_plist().fill_value_defined(). Closes #989, half of #878, companion #3.

Validation: TestBasicEquivalence vlen-string-without-_FillValue turns green.

Phase 4 — Cross-parser consistency

Task	Issues	Approach
Canonical contract	#287, #68	Normative section in docs/custom_parsers.md
Kerchunk audit	#287, #339, #352, #982	Extend hypothesis suite
DMR audit	#904	Possibly out of scope (DMR XML dtype bug)
ZarrParser audit	#811 + audit	Extend hypothesis suite

Validation: new test_*_fill_value_equivalence.py per non-HDF parser.

Phase 5 — Writer-side semantics

Task	Issue	Decision
NaN-in-Parquet	#339	NaN literal vs "NaN" vs null
V2 null preservation	#478	Allow round-trip
NaN padding feature	#22	Out of scope for bugfix track

Phase 6 — Attribute fidelity

Task	Issue	Decision
Non-JSON attribute policy	#715	Drop / cast / error
Scalar dtype preservation	companion #4	Accept widening (doc) or sidecar hint

Test suite

Module	Role	Status
test_zarr_spec_compliance.py	Primary — virtualizarr vs zarr-python, no xarray	✅
test_hdf_fill_value_equivalence.py	HDFParser vs xarray+h5netcdf interop	✅
test_zarr_fill_value_equivalence.py	ZarrParser vs xr.open_zarr interop	✅
test_kerchunk_fill_value_equivalence.py	Next — biggest open-issue payoff	☐
test_tiff_fill_value_equivalence.py	Preventive — GDAL_NODATA	☐
test_icechunk_fill_roundtrip.py	Phase-5 writer round-trip	☐
test_kerchunk_writer_fill_roundtrip.py	Phase-5, catches #339, #478	☐

Shared infra in _fill_value_common.py. Adding a parser = ~150–300 lines (writer helper + open callbacks + strategies + curated examples).

Failure attribution → where the fix lives

Test message	Meaning	Fix in
BothEnginesFailedIdenticallyError	Both engines, same exception	Upstream (xarray)
observed (virtualizarr) failed; reference succeeded	Virtualizarr-specific	Parser
reference failed; observed succeeded	Unexpected — fixture invalid?	Investigate
both engines failed but differently	Multi-engine gap (e.g. vlen arrays)	Per case — usually a clear NotImplementedError guard
Plain assert_identical diff	Both succeed, datasets disagree	Parser

Each runs at raw-attributes (CF decoding off) and decoded-data (CF on).

Future test expansions

Module priority: kerchunk (high — #982, #287, #352) → TIFF (medium, preventive) → FITS (low) → DMR (defer, needs mock OPeNDAP).

Within existing modules: multi-dataset HDF, nested groups, loadable_variables=[...], remote URLs (S3 mock), group-level attributes.

Done: both-engines-fail attribution, hypothesis sharding via @example ids.

References

• Companion: 2026-05-11-hdf5-fill-sentinel-value-fixes.md
• Test branch: test/hdf5-fill-value-property (commit c93abfa)
• Open PR: #988 (fix/problem_fillvalues)
• Label: fill-sentinel-values