Problem
The backfill script populates 1950-2025 but new months need to be added as ERA5-Land data becomes available (~2 month lag). Need an incremental update script that checks what's on S3, downloads only new data, and appends.
Proposed Solution
scripts/pipeline_update.R — reads STAC catalog to determine latest year, fetches new months from CDS, re-aggregates affected years, updates COGs and catalog, pushes to S3. Designed to run in < 6 hours (GitHub Action limit).
Problem
The backfill script populates 1950-2025 but new months need to be added as ERA5-Land data becomes available (~2 month lag). Need an incremental update script that checks what's on S3, downloads only new data, and appends.
Proposed Solution
scripts/pipeline_update.R— reads STAC catalog to determine latest year, fetches new months from CDS, re-aggregates affected years, updates COGs and catalog, pushes to S3. Designed to run in < 6 hours (GitHub Action limit).