Migrate cd_fetch() to DestinE Earth Data Hub Zarr

[NewGraphEnvironment]

Problem

cd_fetch() currently pulls all climate variables from the CDS (ecmwfr) API. Today's experience (#33) and benchmark (#35) showed:

• CDS has aggressive rate limits — sustained ~10 files/hour after the initial ~70-file allowance
• Every request queues server-side and counts against quota, even when polling fails
• The tmax/tmin backfill will take ~3 days of babysitting to complete via CDS
• CDS returns GRIB per-month per-variable, requiring 912 requests per variable for the 1950-2025 backfill

Solution

Migrate cd_fetch() to DestinE Earth Data Hub (EDH) Zarr as the primary data source.

Validated in #35 (benchmark):

• Same product (ERA5-Land, 9 km native, 1950-present)
• Same license (CC-BY 4.0, commercial OK)
• One Zarr store contains all 50 ERA5-Land variables
• 15.9 seconds per month BC bbox vs ~80s via CDS
• 500K requests/month quota — effectively unlimited for our use
• No queueing, no polling, no rate-limit babysitting
• Full backfill ~4 hours unattended vs ~3 days via CDS

Zarr URL: https://data.earthdatahub.destine.eu/era5/reanalysis-era5-land-no-antartica-v0.zarr

Scope

Phase 1: New EDH fetcher alongside CDS

• Add cd_fetch_edh() or refactor cd_fetch() with a source parameter
• Use xarray + zarr via reticulate, OR pure R via stars::read_mdim() (GDAL zarr driver) — evaluate both
• Token via EDH_TOKEN env var (already in ~/.Renviron)
• Variable mapping: EDH t2m → our tmean/tmax/tmin inputs, tp → prcp, d2m → dewpoint, swvl1-4 → soil_moisture, etc.
• Maintain existing output format (monthly COG or intermediate NetCDF) so downstream stages are untouched

Phase 2: Finish the backfill via EDH

• Pull all 912 months for tmax/tmin + any other gaps
• Close tmax/tmin from CDS daily statistics: queue bottleneck and alternatives #33 — backfill is done
• Leave the existing 255 CDS-downloaded files in place if they match EDH output byte-for-byte; otherwise regenerate

Phase 3: Decide on CDS role

• Option A: drop CDS entirely
• Option B: keep CDS as fallback for operational redundancy (both serve the same ERA5-Land data)
• Option C: keep CDS only for near-real-time updates if EDH has more lag than CDS

Phase 4: Update docs and pipeline

• CLAUDE.md — update CDS API section, mention EDH primary
• README + pkgdown — EDH auth setup instructions
• Monthly GitHub Action — switch to EDH
• Secrets — add EDH_TOKEN to repo secrets (rotate the one we've been using in chat)

Out of scope

• Derived variables (vpd, rh) — still computed locally, no source change needed
• Downstream (cd_derive, cd_aggregate, cd_cog_write, cd_stac_catalog, cd_s3_push) — unaffected if intermediate format stays the same

Risks

• EDH reliability / uptime — we're adopting a single provider. Mitigated by keeping CDS as fallback (option B above).
• Zarr chunking may not align with month boundaries, so a "one month pull" could fetch slightly more bytes than needed. In practice fine — quota is generous.
• R zarr tooling is less mature than Python. Reticulate + Python xarray is the pragmatic path; pure R via stars/GDAL is cleaner if it works.

Tracking

Relates to #33 (tmax/tmin operational backfill)
Relates to #35 (alternative source evaluation — SUPERSEDED by this migration)
Relates to NewGraphEnvironment/sred-2025-2026#23

[NewGraphEnvironment]

QA finding 2026-04-12: grid mismatch, and unified-backfill decision

After the EDH-based tmax/tmin backfill completed, ran QA against the existing
CDS-era monthly TIFs and found two real issues plus a strategic clarification.

Finding 1: CDS vs EDH grid misalignment

• EDH-produced tmax/tmin: 120 × 260 grid, extent [-139.95, -113.95, 47.95, 59.95], CRS EPSG:4326
• CDS-produced tmean/prcp/vpd/rh/soil_moisture: 121 × 261 grid (one extra cell each direction), extent shifted 0.1° W and 0.1° N, CRS tag missing

This breaks pixel-wise anomaly computation across variables. An AOI
extracted by cd_extract() would return different pixel values depending
on which variable you query. Can't ship this.

Finding 2: Probe of all EDH variables (Jan 2000 vs CDS Jan 2000)

• t2m → tmean (monthly mean): correlation 0.9947, mean abs diff 0.43 °C — safe drop-in
• d2m → dewpoint: plausible BC winter values, no CDS comparison (we derive VPD/RH from it)
• swvl1..4 → soil_moisture: all four depths match CDS composite range closely — safe
• tp → prcp: naive hourly .sum() gives 720 mm vs CDS 86 mm (8× too high!). ERA5-Land hourly tp has GRIB_stepType=accum — it is a running daily accumulation that resets at 00:00 UTC. Wrestling the accumulation semantics in Python is bug-prone.

Decision: two-Zarr approach for unified backfill

EDH serves three ERA5-Land products. The right split for us:

cd variable	EDH source	Aggregation
tmax, tmin	hourly (reanalysis-era5-land-no-antartica-v0)	daily max/min → monthly mean (already done)
tmean	hourly	monthly mean
dewpoint	hourly	monthly mean (feeds derived vpd/rh)
soil_moisture	hourly, 4 depths	monthly mean per depth, composite in R
prcp	daily (era5-land-daily-utc-v1)	monthly sum of daily totals

Daily product handles the accumulation reset correctly — no need to
reinvent it. State variables stay on hourly since only hourly has all 4
soil moisture depths (daily only has 2).

Next work on this branch

1. Extend scripts/backfill_edh_tmax_tmin.py (or split into a family)
   to cover tmean, dewpoint, soil_moisture, prcp on a unified EDH grid
2. Regenerate all data/backfill/monthly/*_YYYY.tif so the entire
   dataset is internally grid-aligned and CRS-tagged
3. Run R Stage 3 (cd_aggregate → cd_cog_write → cd_stac_catalog) against
   the unified outputs
4. S3 push (with user confirmation)

Scripts added to branch for audit trail

• scripts/qa_monthly.R — grid alignment, time coverage, physical sanity checks across variables
• scripts/probe_edh_vars.py — pulls one month of each EDH variable, compares to CDS, validates aggregation semantics

These stay in the repo as durable QA tooling.