Skip to content

Push ecosystem filters down to the parquet (fix per-document reload)#9

Merged
tylere merged 1 commit into
mainfrom
feat/filter-pushdown
Jul 18, 2026
Merged

Push ecosystem filters down to the parquet (fix per-document reload)#9
tylere merged 1 commit into
mainfrom
feat/filter-pushdown

Conversation

@tylere

@tylere tylere commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Why

EcosystemsGeoParquet.filter() loaded the entire national GeoParquet into memory (~11 GB for the Colombia map) then masked in-memory. Every crit_b document runs filter(code) in its own kernel, so a full render reloaded the whole national file ~85 times — turning the render into a multi-hour job (and it's the remaining bottleneck after the AOO-grid cache fix).

What

  • filter(): for an exact match on a not-yet-loaded parquet, read only the matching rows via pyarrow predicate pushdown (filters=[(col, "==", value)]) instead of load() + mask. Regex and already-loaded data keep the in-memory base behavior. Result columns are unchanged (behavior-identical rows).
  • _read() / _load(): read gs:// through pyarrow's native GCS backend (row-group pruning via range reads, no gcsfs), consistent with _remote_file_exists and the cache read/write path. http(s)/s3/az still use fsspec.

Pushdown only prunes when the parquet is clustered by ecosystem (small row groups). Paired with an ecosystem-sorted gs:// copy of the data, filtering one ecosystem drops from ~11 GB / 60 s to 0.39 GB / 3 s (measured end-to-end through the Ecosystems API).

Verification

  • New TestEcosystemsGeoParquet tests: pushdown avoids full-load (_cached stays None), pushdown result equals full-load + mask (byte-identical rows), regex falls back to in-memory. Written test-first (the no-full-load test fails on the old code).
  • Full test_ecosystems.py + test_aoo.py pass (except the pre-existing TestEcosystemsToRaster libgdal-dylib env failures, unrelated).

🤖 Generated with Claude Code

EcosystemsGeoParquet.filter() loaded the entire national GeoParquet into
memory (~11 GB for the Colombia map) and masked in-memory, so every crit_b
document reloaded the whole file — making a full render take hours.

- filter(): for an exact match on a not-yet-loaded parquet, read only the
  matching rows via pyarrow predicate pushdown (filters=[(col, "==", value)]).
  Regex and already-loaded data keep the in-memory base behavior.
- _read()/_load(): read gs:// through pyarrow's native GCS backend (row-group
  pruning via range reads, no gcsfs), consistent with _remote_file_exists and
  the cache read/write path; http(s)/s3/az still use fsspec.

Effective only when the parquet is clustered by ecosystem (small row groups);
against an ecosystem-sorted gs:// copy, filtering one ecosystem drops from
~11 GB / 60 s to 0.39 GB / 3 s, verified end-to-end.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@tylere
tylere merged commit ae5c633 into main Jul 18, 2026
@tylere
tylere deleted the feat/filter-pushdown branch July 21, 2026 20:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant