Skip to content

Chunked VRT missing_sources='warn' under-reports decode-time holes in attrs['vrt_holes'] #2989

@brendancol

Description

@brendancol

Describe the bug

Reading a VRT with chunks= and missing_sources='warn' does not reliably record decode-time holes in attrs['vrt_holes'].

The chunked path (xrspatial/geotiff/_backends/vrt.py, around line 1160) pre-populates attrs['vrt_holes'] from a parse-time os.path.exists sweep over every source. That sweep only catches sources whose backing file is missing on disk. A source file that exists but is corrupt, truncated, or fails to decode (zlib/zstd codec error, malformed IFD) passes the existence check, so it never lands in attrs['vrt_holes'] at build time.

The eager path finds these holes at decode time. It catches (OSError, ValueError, struct.error) plus the codec decode exceptions per source and appends each to vrt.holes, which read_vrt surfaces as attrs['vrt_holes']. Under chunked dispatch each per-task decode catches its own failure and warns from the worker, but those records cannot be reduced back onto the parent DataArray's attrs.

So a caller can read a chunked VRT under missing_sources='warn', get a DataArray whose vrt_holes attr lists only the missing-file holes (or is absent entirely), branch on "vrt_holes" in da.attrs the way the docstring suggests, then compute and still get a partial mosaic if the compute-time warnings are filtered or missed.

Expected behavior

The chunked attrs['vrt_holes'] contract should be documented as statically-detectable (missing-file) holes only, so callers stop treating the absence of the attr as proof of a complete mosaic. On top of that, a lenient chunked read should tell the caller up front that decode-time codec failures on existing sources surface as per-task warnings at compute and will not appear in attrs['vrt_holes'].

Forcing an eager decode of every source at graph-build time would surface every hole but defeats lazy reading, so that is off the table.

Additional context

  • Eager decode-time hole catch: xrspatial/geotiff/_vrt.py around line 1599.
  • Chunked static sweep and raise guard: xrspatial/geotiff/_backends/vrt.py around line 1160.
  • The existing missing_sources='raise' chunked guard already raises up front for missing files. The same silent-partial-mosaic risk applies to a 'raise' caller whose corrupt-but-existing source only raises inside a worker task that may never run.

Tradeoff

A documentation-only fix is cheap and honest but leaves the gap open. A build-time heads-up warning under 'warn' keeps the lazy contract intact and points the caller at where to look, at the cost of one extra warning on every lenient chunked read that has decode-capable sources. Forcing eager decode is rejected.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingdaskDask backend / chunked arraysgeotiffGeoTIFF modulevrtVRT support contract

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions