Fix gs:// cache detection in _remote_file_exists#8
Merged
Conversation
make_aoo_grid_cached reads and writes gs:// caches via pyarrow's native GCS
backend (gpd.read_parquet / to_parquet), but the existence check used fsspec,
which needs gcsfs. When gcsfs is absent, _remote_file_exists('gs://...')
silently returned False, so a present cache was never detected and the AOO
grid was recomputed from the full national ecosystem map — a multi-GB,
OOM-prone operation that defeats the cache entirely.
Route gs:// through pyarrow.fs.GcsFileSystem (consistent with the read/write
path, no gcsfs dependency); keep fsspec for local and other paths.
Verified against a real gs:// cache: existence now True, make_aoo_grid_cached
returns 12,030 cells in 4.5s at 0.2 GB peak RSS (vs ~11 GB when recomputing).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
make_aoo_grid_cachedreads and writesgs://caches via pyarrow's native GCS backend (gpd.read_parquet/to_parquet), but_remote_file_existschecked existence via fsspec, which needsgcsfs. Whengcsfsisn't installed,_remote_file_exists('gs://…')silently returnsFalse— so a present cache is never detected and the AOO grid is recomputed from the full national ecosystem map (a multi-GB, OOM-prone operation). This is exactly what defeated ags://-hosted AOO grid cache and kept OOM-killing a country render.What
Route
gs://throughpyarrow.fs.GcsFileSystem.get_file_info(consistent with the read/write path, nogcsfsdependency). Local and other paths still use fsspec.Verification
TestRemoteFileExistsunit tests: gs:// uses pyarrow (not fsspec), missing gs:// →False, local paths still work. Written test-first (the gs:// test fails on the old code). Fulltest_aoo.pysuite: 46 passed.gs://cache end-to-end:_remote_file_existsnowTrue, andmake_aoo_grid_cachedreturns 12,030 cells in 4.5 s at 0.2 GB peak RSS (vs ~11 GB when it recomputed).🤖 Generated with Claude Code