geotiff: share lazy finalization helper across dask backends (PR C of #2162)#2205
Conversation
…2162) Both dask read paths had the same 25-line validate-populate-stamp block. Replace each with one call to `_finalize_lazy_read_attrs` from #2177 so a future fix lands in both backends at once. The helper's `dtype` argument is used for two things at once: the resolved graph dtype (for `masked_nodata`) and the caller's cast attr (for `nodata_dtype_cast`). The dask path has to keep those separate, because masking on an integer source auto-promotes to float64 without the caller asking, and that auto-promotion must not leak out as `nodata_dtype_cast`. Fix it up at the call site after the helper returns. Closes #2178. Wave 2 of #2162.
brendancol
left a comment
There was a problem hiding this comment.
PR Review: geotiff: share lazy finalization helper across dask backends (PR C of #2162)
Blockers
None.
Suggestions
-
dask.py:399-404andgpu.py:1587-1591carry an identical 4-line fixup block (if dtype is None: pop / elif nodata: set). Consider pulling it into a small helper in_attrs.py(e.g._apply_caller_dtype_cast(attrs, *, caller_dtype, has_nodata)) so the two backends share the fixup and a future change lands in one place. Not blocking; the current shape is fine. -
_attrs.py:_finalize_lazy_read_attrsdocstring (already on main) flags thedtypeconflation. Worth a follow-up issue to split it intograph_dtype+caller_dtypeso the call-site fixup goes away entirely. Out of scope here.
Nits
-
test_lazy_finalization_parity_2162.py:81declares_BACKEND_MARKER_KEYS: frozenset[str] = frozenset()and_strip_backend_markersbecomes an identity transform. Either drop both and compare attrs directly, or keep the indirection. The comment defends the current shape, but the empty frozenset reads odd at first glance. -
test_lazy_finalization_parity_2162.py:266-275(test_dtype_cast_absent_without_caller_dtype) assertsout.dtype == np.float64on both backends, which is correct, but it does not exercise thedtype.kind != 'f'branch of the call-site fixup (elif nodata_attr is not None: attrs['nodata_dtype_cast'] = np.dtype(dtype).name). Consider adding a case withmask_nodata=False, dtype=np.int32so the integer-cast branch is also pinned.
What looks good
- The helper migration is one-for-one with the pre-helper inline code. Validate-then-populate-then-stamp order is preserved.
nodata_attr(the pre-MinIsWhite-inversion value) is correctly threaded to the helper in the CPU dask path, matching the pre-helper contract forattrs['nodata'].- The
dtypeconflation is recognised and the call-site fixup is the minimal correct fix for the pre-helper contract. - 19 new tests cover the five georef states, both dask backends, and the dtype-cast presence/absence cases. All pass.
- The 91 existing tests in the three pinned regression files (
test_nodata_lifecycle_attrs_2135.py,test_nodata_semantics_split_1988.py,test_georef_status_2136.py) still pass. - Full geotiff test suite passes (4571 tests; the one failure is pre-existing on main and unrelated).
Checklist
- Refactor preserves the pre-helper attrs contract on both dask backends.
- Both implemented backends (dask+numpy, dask+cupy) produce consistent results (full attrs-dict parity test).
- NaN handling unchanged.
- Edge cases covered: 5 georef states + cast/no-cast + int/float fixtures.
- Dask chunk boundaries unchanged (no graph code changed).
- No premature materialization or unnecessary copies (helper does not touch arrays).
- [N/A] Benchmark exists or is not needed: pure refactor.
- [N/A] README feature matrix: no new functions.
- [N/A] Docstrings: no public API change.
- Factor the call-site fixup into `_apply_caller_dtype_cast` in `_attrs.py` so both dask backends share the logic. Follows up the reviewer's note that the two backends carried an identical 4-line block. - Drop the empty `_BACKEND_MARKER_KEYS` indirection in the parity test; compare attrs dicts directly. The future-proofing comment was not paying its weight with an empty frozenset. - Add an integer-cast parity test pinning the `dtype.kind != 'f'` branch of the fixup (`mask_nodata=False, dtype=np.int32` on both backends). Filed #2206 to track splitting the helper's `dtype` parameter into `graph_dtype` + `caller_dtype` so the call-site fixup goes away entirely. Out of scope for this PR per the wave-1 helper signature freeze.
brendancol
left a comment
There was a problem hiding this comment.
Follow-up PR Review: geotiff: share lazy finalization helper across dask backends (PR C of #2162)
Disposition of original findings
- Suggestion 1 (factor fixup helper): Fixed.
_apply_caller_dtype_castlives in_attrs.pynext to_finalize_lazy_read_attrs; both dask backends call it directly. The diff is small and the intent is clearer. - Suggestion 2 (split helper's
dtypeparameter): Deferred to #2206. The helper signature was frozen as part of wave 1; the split is the same idea, just one PR later. - Nit 1 (empty
_BACKEND_MARKER_KEYSfrozenset): Fixed. Removed the indirection; the parity test compares dicts directly. The docstring and one comment that referenced "backend markers" were also updated. - Nit 2 (missing integer-cast branch test): Fixed. Added
test_dtype_cast_records_integer_targetparametrized over both backends. Thedtype.kind != 'f'path is now pinned.
Second-pass findings
None.
What still looks good
- Both dask backends call
_finalize_lazy_read_attrs+_apply_caller_dtype_castin two lines each. The fixup is no longer duplicated. - The new helper docstring covers all three branches (caller_dtype None / set+has_nodata / set+no_nodata) so future readers do not have to derive the contract from the call site.
- 21 parity tests pass (up from 19 after adding the integer-cast case).
- 91 regression tests in the three pinned files still pass.
- 4573 tests pass in the full geotiff suite. The same one pre-existing failure as before remains unrelated.
Checklist
- All blockers fixed.
- All suggestions either fixed or deferred to a tracked follow-up.
- All nits fixed.
- Tests still pass on both backends after the follow-up commit.
Summary
Wave 2 of #2162. Both dask read paths had a 25-line validate-populate-stamp block that was identical except for a couple of comment differences. Replace each with one call to
_finalize_lazy_read_attrsfrom #2177 (now on main) so a future fix lands in both backends at once.xrspatial/geotiff/_backends/dask.py: CPU+dask finalization inread_geotiff_dasknow goes through the helper.xrspatial/geotiff/_backends/gpu.py: GPU+dask finalization in the dask branch ofread_geotiff_gpunow goes through the helper. The three eager GPU sites in this file are out of scope (sibling PR D, GeoTIFF: migrate eager backends to shared eager finalization helper (PR D of #2162) #2179).There is one wrinkle. The helper's
dtypeargument is used for two things at once: the resolved graph dtype (which drivesmasked_nodata) and the caller's cast attr (which drivesnodata_dtype_cast). The dask path has to keep those separate, becausemask_nodata=Trueon an integer source auto-promotes the graph dtype tofloat64without the caller asking, and that auto-promotion must not leak out asnodata_dtype_cast. The migration passes the resolvedtarget_dtype/declared_dtypesomasked_nodatais right, then fixes upnodata_dtype_castat the call site to match the pre-helper contract.Preserved behavior:
attrs['nodata_pixels_present']stays absent on lazy outputs (geotiff: split overloaded masked_nodata into separate nodata lifecycle signals #2135 contract).attrs['nodata_dtype_cast']is recorded only when the caller passeddtype=.dask.py:227-238) still preempts the sentinel before chunk tasks see it;nodata_attr(the pre-inversion value) is what reachesattrs['nodata'].Closes #2178. Depends on #2177 (merged).
Test plan
test_lazy_finalization_parity_2162.py(19 tests) covering:nodata_pixels_presentabsent on both backends.nodata_dtype_castmatches across backends when caller forces a cast.nodata_dtype_castabsent on both backends when the graph dtype was auto-promoted by masking.georef_statusmatches across both backends for full / transform_only / crs_only / none / rotated_dropped.test_nodata_lifecycle_attrs_2135.py,test_nodata_semantics_split_1988.py,test_georef_status_2136.pypass.xrspatial/geotiff/tests/run: 4571 passed, 45 skipped. One pre-existing failure (test_lowlevel_write_pushdown_2138.py::test_write_vs_to_geotiff_byte_parity_uint8[lz4]) reproduces on main and is unrelated.