Evaluate alternative ERA5-Land hourly data sources

[NewGraphEnvironment]

Question

Is there a better source than CDS for ERA5-Land hourly 1950-2025 over BC?

Why this data and resolution matter

ERA5-Land is the gold-standard reanalysis dataset for land-surface climate. What we need is unique because:

1. Hourly resolution at 9 km grid spacing, 1950-present

• No other public dataset combines that temporal depth (76 years) with that spatial fidelity over land
• Daily-only products lose the diurnal cycle — cannot compute true daily max/min, evapotranspiration, or fire weather indices without hourly data
• Most "long records" are station-based with huge spatial gaps; ERA5-Land fills them physically-consistently

2. Reanalysis, not interpolation

• ECMWF runs a frozen state-of-the-art weather model and assimilates billions of observations (stations, satellites, radiosondes, ships, aircraft) into a coherent global state every hour
• Where stations exist, it is anchored to them. Where they do not (most of BC's wilderness), it is still physically realistic
• Climate departure analysis needs a baseline that is gridded and complete — station data has gaps and biases that fake "trends" when stations open/close

3. Why tmax/tmin specifically

• Daily max/min is what ecosystems actually feel — fish thermal stress, vapor pressure deficit, snowmelt timing, fire weather, frost dates
• Mean temperature hides the extremes that drive ecological response
• Trend in tmin (overnight lows warming faster than daytime highs) is one of the clearest climate-change fingerprints — losing it would gut the analysis

4. Full-territory backfill enables the cd package to ship

• Consumer-side baseline/anomaly/trend computation requires the raw monthly time series on STAC
• Without 1950-2025 coverage, users cannot pick custom reference periods (e.g., 1961-1990 WMO baseline vs. 1981-2010)
• Once on S3 as COGs, anyone running cd_extract() for any AOI in BC gets full historical departure analysis in seconds — no CDS account, no batch downloads

5. Grid resolution matters for our work

• 9 km is fine enough to resolve valley-bottom vs. ridgetop climate in places like the Bulkley, Skeena, Parsnip
• For fish habitat, riparian restoration planning, and species range work, anything coarser smears out exactly the gradients that matter
• Most "downscaled" products are interpolations of coarser data and add false confidence — ERA5-Land is the actual model output

Bottom line

76 years × 12 months × hourly × full BC bbox at 9 km = the substrate for every climate-context section in every report we will write for the next decade. Worth getting right.

Evaluation criteria (must-haves)

• ERA5-Land specifically (NOT ERA5) — 9 km native resolution. ERA5 parent is ~31 km and would be a downgrade.
• Hourly temporal resolution (so we can compute true daily max/min)
• Coverage: 1950-present
• Programmatic access (no manual download)
• Reasonable quota for one-time backfill (~700 files left as of 2026-04-11)

Candidates to evaluate

• DestinE Earth Data Hub — confirm whether it carries ERA5-Land at 9 km native, or only ERA5 at 31 km
• Google Earth Engine — has ERA5-Land hourly. Check quota/license for non-academic commercial use
• AWS Open Data — ERA5 mirror exists. Check ERA5-Land hourly availability
• WeatherBench / ARCO-ERA5 — research mirrors; verify product and resolution
• Second CDS account — boring fallback, but works
• CDS support email — explain rate-limit history, request quota bump given legitimate scientific use case

Findings

Appended as we research.

---

Relates to #33 (operational backfill saga via CDS)
Relates to NewGraphEnvironment/sred-2025-2026#23

[NewGraphEnvironment]

Research findings 2026-04-12

Comparison

Source	Product	Resolution	Temporal	Access	Commercial OK
DestinE Earth Data Hub	ERA5-Land	9 km native (confirmed)	1950-present (unconfirmed; some mirrors truncate to 1979)	Zarr via HTTPS, xarray/kerchunk	Yes (CC-BY 4.0) — free tier with unconfirmed rate limits
Google Earth Engine	ECMWF/ERA5_LAND/HOURLY	9 km native (~0.1°)	1950-01-02 to present	GEE Python/JS API	No — commercial use requires paid license
AWS Open Data (era5-pds)	ERA5 (parent, NOT Land)	31 km	1979-present	S3 Zarr/NetCDF	Yes — but wrong product
ARCO-ERA5 (GCS)	ERA5 (parent, NOT Land)	31 km	1940-present	Zarr on gs://gcp-public-data-arco-era5/	Yes — but wrong product
WeatherBench 2	ERA5 regridded for ML	0.25° / 1.5° / 5.625°	1959-2022	Zarr on GCS	Yes — but wrong product
CDS-beta	Same as current CDS	Same	Same	Same ecmwfr API	Yes — same backend, same rate limits

Key finding

Only two sources actually carry ERA5-Land at 9 km hourly: DestinE and GEE. Everything else is ERA5 (31 km parent), which is a resolution downgrade we are not willing to accept.

GEE is blocked by licensing — commercial/consulting use requires a paid license (quote-based, typically thousands/yr).

DestinE is the candidate. Zarr access means one dataset-open + time slice instead of 912 separate requests. Night-and-day difference from CDS if it works out.

Recommended next steps

1. Spend 1-2 hours on DestinE verification:

   • Create account at platform.destine.eu
   • Read commercial-use terms (confirm consulting is OK)
   • Confirm ERA5-Land Zarr goes back to 1950 (not truncated to 1979)
   • Pull one test month for BC bbox via xarray — benchmark throughput
   • Check free-tier quota vs. our total volume

2. If DestinE does not pan out: email copernicus-support@ecmwf.int explaining the one-time 912-file backfill for a legitimate commercial/scientific project. ECMWF grants temporary quota bumps — faster than 3 days of babysitting CDS, zero code changes.

3. Fallback if both fail: continue babysitting CDS at ~10 files/hour sustained. It works; it is just slow.

What we are NOT doing

• AWS / ARCO / WeatherBench — wrong product (31 km ERA5, not 9 km ERA5-Land)
• GEE — commercial license required
• CDS-beta — same rate limits as current CDS

---

Relates to #33

[NewGraphEnvironment]

Benchmark result 2026-04-12: EDH is the winner

Pulled one month (1960-01) of hourly t2m for BC bbox from DestinE Earth Data Hub via Zarr.

Result: 15.9 seconds end-to-end. Zarr open in 2.9s, subset + download + materialize in 15.9s. Values match reality (-45 to +13 °C for BC January, 34% ocean NaN as expected).

Comparison

Metric	CDS (current)	EDH Zarr
Per-month pull	~80s (w/ polite sleep)	~15s
Full backfill (912 months)	~3 days babysitted	~4 hours unattended
Rate limiting	Aggressive, ~10 files/hr sustained	500K requests/month quota
Queueing	Every request queues server-side	None — Zarr is chunk HTTP
Variables accessible	1 per request	All 50 ERA5-Land variables in one store
Auth	CDS account + UID/key	DestinE PAT (single token)
Temporal coverage	1950 to near-present	1950-01 to 2026-02 (monthly updates)
Commercial license	CC-BY	CC-BY (same source)

Validated

• DestinE EDH carries ERA5-Land at native 9 km (0.1° grid, 1472 × 3600 global)
• 1950-01 start date confirmed (not truncated to 1979)
• Temporal range: 1950-01-01 to 2026-02-28 (667,632 hourly timesteps)
• All ERA5-Land variables present: asn, d2m, e, es, evabs, evaow, evatc, evavt, fal, lai_hv, lai_lv, lblt, licd, lict, lmld, lmlt, lshf, ltlt, pev, ro, rsn, sd, sde, sf, skt, slhf, smlt, snowc, sp, src, sro, sshf, ssr, ssrd, ssro, stl1, stl2, stl3, stl4, str, strd, swvl1, swvl2, swvl3, swvl4, t2m, tp, tsn, u10, v10
• Commercial use permitted (CC-BY 4.0)
• 500K requests/month quota confirmed per EDH docs
• Zarr URL stable and public: https://data.earthdatahub.destine.eu/era5/reanalysis-era5-land-no-antartica-v0.zarr
• Authentication works via inline PAT in URL (edh:<token>@...)

Implications beyond tmax/tmin

This is not just an alternative for the tmax/tmin backfill — it is a replacement for the entire CDS-based cd_fetch() producer pipeline. All 7 of our climate variables (tmean, tmax, tmin, prcp, dewpoint, soil_moisture 4 layers, rh/vpd derivable) can pull from the same Zarr store. One auth, one data source, no queueing.

Test script

scripts/test_edh_era5_land.py — portable via PEP 723 (uv run scripts/test_edh_era5_land.py, no venv setup).

Recommendation

Switch cd_fetch() to EDH Zarr as the primary data source. Keep CDS (ecmwfr) as a fallback for operational redundancy — same underlying data, same license. File a new issue to plan the migration.

Relates to #33

[NewGraphEnvironment]

Resolved by v0.1.0 / #36 — producer migrated from Copernicus CDS to DestinE Earth Data Hub (EDH). EDH was the leading candidate in this issue's evaluation matrix and has been in production since April 2026 with no rate limiting and ~5x faster fetches. See NEWS.md v0.1.0 entry.