Integration testing strategy for pcmdi_diags

[forsyth2]

@zhangshixuan1987 @chengzhuzhang I'm trying to figure out the best way to test pcmdi_diags in zppy. These are my thoughts based on the email thread related to #845 (reply in thread):

zppy usually handles testing code changes in underlying packages by specifying which dev environment to use via the environment_commands parameter. Through that mechanism we can use a precise version of e3sm_to_cmip, e3sm_diags, mpas_analysis, zppy-interfaces (for global_time_series and pcmdi_diags). Changing the NCO version (for climo and ts) is a little trickier, involving a separate parameter nco_path. And then some packages (ilamb, livvkit) we just have to rely on whatever the latest released version is, because it's an independent package not maintained by the project and/or there hasn't yet been a need to use a different version.

Notice the usual trend is that a zppy task calls a Python package. Now, these packages can and do rely on third-party libraries/packages, but the core work we're interested in for integration testing is done in those packages. However, zppy-interfaces's pcmdi_diags functionality is actually an intermediary element between zppy and pcmdi_metrics. This design was chosen because wrapping pcmdi_metrics directly from zppy was deemed overly complex (a decision I still think was right, considering pcmdi_diags has a very large/complex codebase). What this means though is that when we test a dev environment, we're testing a dev environment for this intermediary element (zppy-interfaces's pcmdi_diags), not a dev environment for the latest pcmdi_metrics package itself.

This now leads us to several options for improving testing of pcmdi_diags, given below, from easiest to most difficult to implement:

1. Keep it as-is. That is, we keep treating pcmdi_metrics as if it were just any other third-party package we use.
   • For the diffs we're currently seeing, this option would mean I'd just call the diffs acceptable, and update the expected results. Then, once pcmdi_metrics updates and new diffs appear, I'd' update the expected results again.
2. Keep it as-is but treat pcmdi_metrics special. That is, if we notice bugs in it, don't discount them.
   • For the diffs we're currently seeing, this option would mean I'd wait for a new pcmdi_metrics version to be released, and then re-run the test, and assuming any new diffs that run produces are acceptable, I'd update the expected results to match those. That is, this differs from option 1 because the expected results don't get updated until pcmdi_metrics is fixed.
3. For ILAMB, we just always use the Unified environment to test it, because we don't have access to latest dev/unreleased environments. pcmdi_metrics is in a similar situation -- we use it, but we're not the maintainer (PCMDI is, not E3SM). It would follow that we should pin pcmdi_metrics to a specific version inside zppy-interfaces dev environment. It's currently pinned only as pcmdi_metrics>=3.9.3. That way changes that appear in testing are going to be solely because of changes in zppy-interfaces's pcmdi_diags, not the PCMDI-maintained pcmdi_metrics package. (This still isn't completely true, because there's still variation in the version of third-party libraries like matplotlib. For example, if the diffs we're seeing are bugs in how pcmdi_metrics interacts with matplotlib, then this option would avoid showing us bugs outside our control. If however, the matplotlib version changes how plots look independently of pcmdi_metrics's version, the diffs would still show up).
   • For the diffs we're currently seeing, this option would mean I'd re-run the test using a zppy-interfaces dev environment with the pinned pcmdi_metrics (v4.0.4), and assuming any new diffs that run produces are acceptable, I'd update the expected results to match those.
4. Add a new parameter to zppy, an analog to environment_commands called something like pcmdi_metrics_version and then we build into zppy/templates/pcmdi_diags.bash a line to specifically install a specific pcmdi_metrics version after running the regular environmnet_commands.
   • For the diffs we're currently seeing, this option would mean I'd re-run the test using this new parameter to pin pcmdi_metrics at Unified's version (v4.0.4), and assuming any new diffs that run produces are acceptable, I'd update the expected results to match those.

I think option 3 (pin pcmdi_metrics in zppy-interfaces's dev environment) makes the most sense, but please let me know what you think. I think option 4 (adding a new parameter) is probably too complicated.

For context, keeping the expected results up-to-date (by updating them often) is very important. If too many code changes go in between expected results updates it can be hard to tell which diffs are causing problems. For example, right now global_time_series has a large number of diffs because the plots are all slightly shifted (presumably due to matplotlib version used). That means it's difficult for me to tell if new PRs (like E3SM-Project/zppy-interfaces#53) are causing diffs themselves.

[zhangshixuan1987]

@forsyth2: Thanks, Ryan, for laying out these options. I agree that option 3 makes the most sense, and option 4 seems unnecessarily complicated unless users have a broader need to select the pcmdi_metrics version through zppy.

If feasible, could we make option 3 even more controlled by running the reference and candidate versions of zppy/zppy-interfaces using the same base environment definition? Ideally, we would create or clone two environments from the same resolved YAML or lock file and vary only the versions of zppy and zppy-interfaces being tested. This would intentionally eliminate the possibility that observed differences are caused by changes in pcmdi_metrics, matplotlib, or other dependency libraries.

Using the same YAML file would help, although cloning the same resolved environment or using a lock file would provide a stronger guarantee because solving the YAML at different times could still produce different dependency versions.

For the current diffs, perhaps we could:

1. Establish a common environment with pcmdi_metrics==4.0.4 and fixed versions of the other dependencies.
2. Run both the reference and candidate zppy/zppy-interfaces versions in copies of that environment.
3. Review any remaining differences, which should then be attributable primarily to changes in zppy or zppy-interfaces.
4. If those differences are acceptable, update the expected results using this environment as the new baseline.

We could still test periodically against the latest Unified environment or a newer pcmdi_metrics release to identify upstream compatibility issues, but that could be separate from the controlled regression test used to evaluate zppy and zppy-interfaces changes.

[chengzhuzhang]

Thanks for clarifying. I see the distinction now. I was thinking about whether we should take responsibility for testing the third-party packages themselves, but I agree that’s not the goal. The goal is to keep the testing environment stable enough that changes in those dependencies don’t obscure changes in the packages we actually want to test.

Using the recent E3SM Diags testing as an example, it also uses a defined environment for its integration/image-regression testing, and dependency changes can otherwise show up as image differences.

I think I’m leaning toward option 3 as well, pinning pcmdi_metrics in the zppy-interfaces development environment seems like a reasonable way to keep the scope of the test focused without adding another changes.

[forsyth2]

Sounds good, thanks!

[forsyth2]

If it's overly cumbersome or not working properly

This is in fact proving somewhat cumbersome and the diff required for it is adding substantially to #774. There are also two cases of "use a frozen environment" to concern ourselves with:

1. Easier: if the current "expected results" were generated using dev environments, then we just need to freeze the versions used there.
2. Harder: if the current "expected results" were generated using the Unified release (this is the current case), then we need to freeze a dev environment that matches up with Unified as much as possible. This is proving particularly tricky. Unified is built rather differently after all, including using pixi rather than conda.

@chengzhuzhang Personally, I think the steps below would be the best course of action (this updates my original plan detailed here). Do you agree and/or have further suggestions? Thanks!

1. Review & merge the current testing script, Add main branch testing automation #774 (as it was before the commits to freeze envs). We know from multiple runs on Testing zppy development for E3SM Unified 1.14.0 (Fall 2026) #845 now that the testing script is working as intended. We don't necessarily need to have it giving 100% passing tests to know this, so we don't necessarily need to have the "expected results" updated before merging it.
2. Move the environment freezing commits to a new PR and try to get that working in at least the easier case listed above, and maybe the harder case as well. Once we have a good frozen environment base, we can go about updating the expected results.
3. Implement New post-v3.2.0 test cfgs #837
4. Once we have all that working, then we can check if the test script is actually usable by anyone except me -- e.g., by having you or another developer run it.

[chengzhuzhang]

Thanks @forsyth2.
Yes, let's finish item 1.
Item 2 is nice-to-have but its priority is probably low, given the piled up tasks on hands.
Item 3 can wait and we still need EAMxx output to stabilize and probably need more iterations on zppy support, the group is considering update vertical level, etc. and we need some flexibility for supporting new updates.
Item 4: yes I'd like to run the auto testing framework.

[forsyth2]

@chengzhuzhang Thanks, I'll get started on that plan then!