Fix _parse_rat to pad category_names to pixel indices for sparse RATs#3643
Merged
Conversation
…array-contrib#3591) When a sidecar's RAT has a sparse or non-zero-based Value column (e.g. values [1, 2, 5] from QGIS/GDAL), _parse_rat now pads category_names and category_colors with empty strings / (0,0,0,0) so list index equals pixel value, matching the attrs contract. Fails closed on negative values and values >= 1,000,000 to avoid OOM from pathological RATs.
Melissari1997
commented
Jul 7, 2026
Melissari1997
left a comment
Collaborator
Author
There was a problem hiding this comment.
PR Review: Fix _parse_rat to pad category_names to pixel indices for sparse RATs
Blockers (must fix before merge)
None.
Suggestions (should fix, not blocking)
-
_pam.py:241-242--max(r[0] for r in rows)/min(r[0] for r in rows)iteraterowstwice.rowsis typically small (a handful of RAT classes), so this is negligible in practice, but computing both in a single pass overrows(e.g. manual loop or tee) would be trivially cleaner. Not blocking.
Nits (optional improvements)
-
_pam.py:194--_MAX_CATEGORIES = 1_000_000is module-level but_parse_ratis the only consumer. If you prefer keeping the guard close to the code it protects, it could be a local constant inside_parse_rat. Style preference, no functional difference.
What looks good
- The fix correctly aligns the output with the documented
category_namescontract ("index == pixel value"). - Fails closed on the two known OOM hazards (negative values, pathological max) rather than silently allocating.
- The
_MAX_CATEGORIESguard (1M) gives ample headroom for realistic classified rasters while preventing genuine OOM. - Six well-structured tests cover: the exact repro from #3591, color padding, negative guard, max guard, no-Value-column path with sparse Row indices, and regression on dense 0-based RATs.
- All 10 tests in the file pass, including the 4 pre-existing reader-edge tests.
- The change is surgical -- only
_parse_ratis modified;build_pam_xml,read_pam_sidecar, and callers are untouched. - Backend coverage is correctly described as backend-agnostic (runs during
open_geotiffmetadata attach on all backends).
Checklist
- Algorithm matches reference/paper -- N/A (metadata parser fix, not a numerical algorithm)
- All implemented backends produce consistent results -- N/A (backend-agnostic metadata reader)
- NaN handling is correct -- N/A (no numerical computation)
- Edge cases are covered by tests -- negative values, huge values, sparse values, no Value column, dense 0-based regression all tested
- Dask chunk boundaries handled correctly -- N/A
- No premature materialization or unnecessary copies -- no
.values/.compute()calls - Benchmark exists or is not needed -- not needed for a metadata parser
- README feature matrix updated -- N/A (no new public API)
- Docstrings present and accurate --
_parse_ratdocstring updated to reflect new padding behavior
Melissari1997
commented
Jul 7, 2026
Melissari1997
left a comment
Collaborator
Author
There was a problem hiding this comment.
Follow-up review: disposition of findings
Suggestion: single-pass min/max (_pam.py:241-242)
- Fixed in 53d4a2f. Replaced
max()/min()with a singlefor r in rows:loop computing both in one pass.
Nit: _MAX_CATEGORIES module-level vs function-local (_pam.py:194)
- Dismissed. Module-level constants are the established convention in this file:
_USAGE_MINMAX,_USAGE_NAME,_USAGE_RED,_USAGE_GREEN,_USAGE_BLUE,_USAGE_ALPHA,_TYPE_INT,_TYPE_STRINGare all at module level (lines 24-32). Keeping_MAX_CATEGORIESalongside them is consistent with existing style.
All 10 tests continue to pass.
Contributor
|
@Melissari1997 wooohooo |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3591.
When a PAM sidecar's thematic RAT has sparse or non-zero-based values (e.g.
[1, 2, 5]from QGIS/GDAL classified rasters),_parse_ratwas sorting rows and returning a dense list, misaligning category names against their pixel values. The attrs contract promises index == pixel value.category_nameswith empty strings andcategory_colorswith(0,0,0,0)for gaps between sparse RAT rows so list index == pixel value.Backend coverage: backend-agnostic — runs during
open_geotiffmetadata attach on all backends (numpy, dask, cupy, dask+cupy).Test plan
read_pam_sidecar).test_field_defn_without_usage_is_skippedstill passes.