Push ecosystem filters down to the parquet (fix per-document reload)#9
Merged
Conversation
EcosystemsGeoParquet.filter() loaded the entire national GeoParquet into memory (~11 GB for the Colombia map) and masked in-memory, so every crit_b document reloaded the whole file — making a full render take hours. - filter(): for an exact match on a not-yet-loaded parquet, read only the matching rows via pyarrow predicate pushdown (filters=[(col, "==", value)]). Regex and already-loaded data keep the in-memory base behavior. - _read()/_load(): read gs:// through pyarrow's native GCS backend (row-group pruning via range reads, no gcsfs), consistent with _remote_file_exists and the cache read/write path; http(s)/s3/az still use fsspec. Effective only when the parquet is clustered by ecosystem (small row groups); against an ecosystem-sorted gs:// copy, filtering one ecosystem drops from ~11 GB / 60 s to 0.39 GB / 3 s, verified end-to-end. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
EcosystemsGeoParquet.filter()loaded the entire national GeoParquet into memory (~11 GB for the Colombia map) then masked in-memory. Everycrit_bdocument runsfilter(code)in its own kernel, so a full render reloaded the whole national file ~85 times — turning the render into a multi-hour job (and it's the remaining bottleneck after the AOO-grid cache fix).What
filter(): for an exact match on a not-yet-loaded parquet, read only the matching rows via pyarrow predicate pushdown (filters=[(col, "==", value)]) instead ofload()+ mask. Regex and already-loaded data keep the in-memory base behavior. Result columns are unchanged (behavior-identical rows)._read()/_load(): readgs://through pyarrow's native GCS backend (row-group pruning via range reads, nogcsfs), consistent with_remote_file_existsand the cache read/write path.http(s)/s3/azstill use fsspec.Pushdown only prunes when the parquet is clustered by ecosystem (small row groups). Paired with an ecosystem-sorted
gs://copy of the data, filtering one ecosystem drops from ~11 GB / 60 s to 0.39 GB / 3 s (measured end-to-end through theEcosystemsAPI).Verification
TestEcosystemsGeoParquettests: pushdown avoids full-load (_cachedstaysNone), pushdown result equals full-load + mask (byte-identical rows), regex falls back to in-memory. Written test-first (the no-full-load test fails on the old code).test_ecosystems.py+test_aoo.pypass (except the pre-existingTestEcosystemsToRasterlibgdal-dylib env failures, unrelated).🤖 Generated with Claude Code