Fetch PMP obs4MIPs reference data from ESGF and drop unused ERA datasets#799
Conversation
Five reference datasets used by the PMP diagnostics are published on ESGF under obs4MIPs, but were only reachable via the obs4REF registry. Add a fetch request for each so they can be sourced from ESGF directly. One request per source_id: the ESGF search intersects source_id and variable_id, so a combined request would ask each source for variables it does not have. 20CR is published on ESGF as 20CR-V2. Fetching it here makes the data available but does not on its own satisfy the psl modes-of-variability requirements, which still filter on 20CR.
No diagnostic requests ERA-20C or ERA-INT under any source type, so the 22 files each registry carried for them were never fetched for a solve. ERA-20C is not published on ESGF at all, and the ERA-interim record there holds only ua and va, so neither could migrate to obs4MIPs. Remove them from the obs4REF and sample-data registries, drop the now-unreachable ERA-INT entry from the ENSO observation name mapping, and regenerate the affected catalog regression fixtures. ERA-5 is untouched and is still required by four ESMValTool diagnostics. Developers with an existing tests/test-data/sample-data directory need `fetch_sample_data(force_cleanup=True)`: fetch_all_files does not prune files dropped from the registry, and SAMPLE_DATA_VERSION is an upstream tag so it cannot be bumped to force a clean re-fetch.
|
Warning Review limit reached
Next review available in: 21 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughECMWF ERA-20C and ERA-INT entries are removed from reference registries, sample data, and test catalog fixtures. The ENSO remapping is updated, and the ESGF fetch script adds new Obs4MIPs requests, retry handling, outcome reporting, and tests. ChangesERA-INT/ERA-20C removal and new obs4MIPs requests
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Both fetch methods wrapped their search and download in a bare `except Exception` and returned an empty dict, so run_request printed "0 datasets" and main exited 0. A transient ORNL THREDDS 504 was therefore indistinguishable from a request that legitimately matched nothing. Let the fetch methods raise, and classify the result in run_request. NoSearchResults means the search matched nothing, which is reported but is not a failure. Anything else means the datasets exist but could not be retrieved, which fails the run. Retry DatasetLoadError, StalledDownload and GlobusTransferError with an exponential backoff, since ESGF data nodes return intermittent 5xx responses. main now exits non-zero if any request failed, and logs a summary of what succeeded, matched nothing, and failed.
Derive NUM_OBS4REF_FILES from dataset_registry_manager["obs4ref"].registry instead of hardcoding it, so the count can't drift out of sync with obs4ref_reference.txt again. The ERA-20C/ERA-INT removal trimmed the registry from 83 to 61 entries. Regenerate the obs4MIPs ts dataset hash regression fixture. ERA-20C and ERA-INT both provided ts, so removing them legitimately changes the hash of the ts dataset collection in the sample data catalog.
Codecov Report✅ All modified and coverable lines are covered by tests.
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
1 participant
Description
Sources the obs4MIPs reference data that PMP diagnostics need from ESGF, and removes two reference datasets that nothing uses.
Fetch PMP reference data from ESGF
Five datasets requested by the PMP diagnostics are published on ESGF under obs4MIPs but were only reachable via the obs4REF registry.
scripts/fetch-esgf.pynow has a request for each. There is one request persource_idbecause the ESGF search intersectssource_idandvariable_id, so a combined request would ask each source for variables it does not have.pmp-enso-gpcp-obs4mipsGPCP-Monthly-3-2prpmp-enso-tropflux-obs4mipsTropFlux-1-0hfls hfss tauu tspmp-enso-ceres-obs4mipsCERES-EBAF-4-2rlds rlus rsds rsuspmp-hadisst-obs4mipsHadISST-1-1tspmp-modes-20cr-obs4mips20CR-V2pslEvery request was run end to end. Each file was opened with xarray to confirm the expected variable and a sane time axis (TropFlux 463 months, CERES 283, HadISST 1861, 20CR 1704).
20CRis published on ESGF as20CR-V2. Fetching it makes the data available, but the five psl modes-of-variability diagnostics still filter on20CRand will keep resolving from the obs4REF registry. Wiring them up means changing the filter and re-minting those baselines, sincereference_source_idis in that diagnostic'sfacetstuple. Left for a follow-up.Remove unused ERA-20C and ERA-Interim
No diagnostic requests
ERA-20CorERA-INTunder any source type — checked by enumerating the obs4MIPs and PMPClimatologysource_idfilters across every registered diagnostic in all four providers, not by grep. Neither could migrate to obs4MIPs anyway:ERA-20Cis not on ESGF at all, and theERA-interimrecord there is a single 2016-era dataset holding onlyuaandva.That removes 22 entries from each of the obs4REF and sample-data registries, the now-unreachable
ERA-INTentry from the ENSO observation name mapping, and 1496 lines from three catalog regression fixtures, which were regenerated against a freshly fetched sample-data tree.ERA-5is untouched — it is a different product and is still required by four ESMValTool diagnostics.ERA-INT was the only obs4REF sample carrying pressure levels (
vertical_levels: 19), but theplevbranch ofread_vertical_levelshas direct synthetic-file coverage intest_netcdf_utils.py, so no parser coverage is lost.Note for reviewers
fetch_all_filesdoes not prune files that have been dropped from a registry, andfetch_sample_dataonly wipes its output directory whenSAMPLE_DATA_VERSIONchanges — and that version is a tag of the upstreamref-sample-datarepo, so it cannot be bumped here to force a clean re-fetch. Anyone with an existingtests/test-data/sample-data/will still have the 22 ERA files on disk, and theobs4mips_catalog_localregression will fail for them while CI passes. Run this once:Worth fixing durably at some point, either by having
fetch_all_filesprune, or by keying the cleanup on a hash of the registry contents rather than the version tag.Fail the fetch when a request could not be retrieved
Both
fetchmethods wrapped their search and download in a bareexcept Exceptionand returned{}, sorun_requestprinted0 datasetsandmain()exited 0. A transient ORNL THREDDS 504 during verification produced exactly that, and a retry downloaded the file fine. In CI or an unattended bulk fetch, a genuine failure was indistinguishable from a dataset that legitimately has no results.The fetch methods now raise, and
run_requestclassifies the outcome:NoSearchResults— the search matched nothing. Reported, but not a failure.DatasetLoadError,StalledDownload,GlobusTransferError— the datasets exist but could not be retrieved. Retried with an exponential backoff (--max-attempts,--retry-delay), then failed.main()logs a summary and exits non-zero if any request failed.Covered by
tests/scripts/test_fetch_esgf.py(10 tests, no network). Each test was checked against a deliberately mutated script to confirm it fails when the fix is removed: dropping the non-zero exit breaks 2, makingNoSearchResultsretryable breaks 1, and removing the retry loop breaks 2.Checklist
Please confirm that this pull request has done the following:
tests/scripts/test_fetch_esgf.py(10 tests). Three catalog regression fixtures were also regenerated (1237 passed inclimate-ref, 55 inclimate-ref-pmp).docs/getting-started/02-download-datasets.mdpoints at the obs4REF registry, whose contents are updated in place.changelog/—799.feature.md,799.improvement.mdand799.fix.md.Summary by CodeRabbit
New Features
Bug Fixes