Skip to content

Conversation

@nocollier
Copy link
Contributor

@nocollier nocollier commented Jan 23, 2025

Description

Adds the ILAMBStandard Metric to the ILAMB provider that runs our standard analysis. A few comments:

  • ILAMB works in 2 phases. The first of those operates on all pairs of (reference_data, model output) for any given variable. This generates some intermediate files containing scalars we want to show (errors and scores, etc) as well as things we want to plot (netcdf file).
  • In a second phase, we take all this intermediate data and now can compute plot limits and make plots/generate html.
  • For now, I have lumped phase 1 and 2 in the same Metric.run().
  • ILAMB specific reference data has been added as a DataCollection via a pooch registry. This already addresses ILAMB-specific reference datasets #69.
  • For the moment, this will only perform the bias component of the analysis. Need to expand ilamb3 once this interface is completely ironed out.
  • I have some un-merged work on ilamb3 currently that is needed to make Phase 2 work completely.

Checklist

Please confirm that this pull request has done the following:

  • Tests added
  • Documentation added (where applicable)
  • Changelog item added to changelog/

@nocollier nocollier marked this pull request as draft January 23, 2025 21:40
@nocollier nocollier self-assigned this Jan 23, 2025
@lewisjared
Copy link
Contributor

lewisjared commented Jan 24, 2025

I have some un-merged work on ilamb3 currently that is needed to make Phase 2 work completely.

FYI: You can use a branch as a dependency https://docs.astral.sh/uv/concepts/projects/dependencies/#git

@nocollier
Copy link
Contributor Author

nocollier commented Jan 27, 2025

This now implements the full metric. I have downloaded some local (historical/land-hist) data to illustrate the grouping works correctly. I get the following when I run ref --solve:

/home/nate/.config/cmip_ref/tmp/ilamb/
└── ilamb_standard_tas_cru4.02
    ├── 6cdca8780b5d843db6900e05fecdea14b4e82358
    │   ├── CanESM5-r1i1p1f1-gn.csv
    │   ├── CanESM5-r1i1p1f1-gn.nc
    │   ├── CanESM5-r1i1p1f1-gn_None_bias.png
    │   ├── CanESM5-r1i1p1f1-gn_None_biasscore.png
    │   ├── CanESM5-r1i1p1f1-gn_None_mean.png
    │   ├── CESM2-r1i1p1f1-gn.csv
    │   ├── CESM2-r1i1p1f1-gn.nc
    │   ├── CESM2-r1i1p1f1-gn_None_bias.png
    │   ├── CESM2-r1i1p1f1-gn_None_biasscore.png
    │   ├── CESM2-r1i1p1f1-gn_None_mean.png
    │   ├── output.json
    │   ├── Reference.nc
    │   └── Reference_None_mean.png
    └── a83768dcad7f86356ade937483bc9d409aa116bc
        ├── CESM2-r1i1p1f1-gn.csv
        ├── CESM2-r1i1p1f1-gn.nc
        ├── CESM2-r1i1p1f1-gn_None_bias.png
        ├── CESM2-r1i1p1f1-gn_None_biasscore.png
        ├── CESM2-r1i1p1f1-gn_None_mean.png
        ├── output.json
        ├── Reference.nc
        └── Reference_None_mean.png

which is the behavior I expected. I have many things still to polish but before the PR is ready I need to:

  • Create some small tests
  • Populate the cmec bundle

I just wanted you @lewisjared to see how I was going about the integration of the metrics.

@nocollier nocollier marked this pull request as ready for review January 30, 2025 19:53
@codecov
Copy link

codecov bot commented Jan 30, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Files with missing lines Coverage Δ
...trics-ilamb/src/cmip_ref_metrics_ilamb/__init__.py 100.00% <100.00%> (ø)
...etrics-ilamb/src/cmip_ref_metrics_ilamb/example.py 100.00% <ø> (ø)
...trics-ilamb/src/cmip_ref_metrics_ilamb/standard.py 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@nocollier nocollier changed the title WIP: Add ILAMB standard analysis Add ILAMB standard analysis Jan 30, 2025
@nocollier nocollier requested a review from lewisjared January 30, 2025 20:59
@lewisjared lewisjared merged commit 6e00b29 into main Jan 31, 2025
13 checks passed
@lewisjared lewisjared deleted the ilamb-data-collection branch January 31, 2025 02:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants