Reorder difference so CSET does $OTHER - $BASE

[James Warner (jwarner8)]

Differences should be test minus control, this PR addresses #2046

Contribution checklist

Aim to have all relevant checks ticked off before merging. See the developer's guide for more detail.

• Documentation has been updated to reflect change.
• New code has tests, and affected old tests have been updated.
• All tests and CI checks pass.
• Ensured the pull request title is descriptive.
• Ensure rose-suite.conf.example has been updated if new diagnostic added.
• Conda lock files have been updated if dependencies have changed.
• Attributed any Generative AI, such as GitHub Copilot, used in this PR.
• Marked the PR as ready to review.

[github-actions]

[James Warner (jwarner8)]

Handing over to David Flack (@daflack) to determine what needs doing to SSIM before review

[David Flack (daflack)]

Notes for SSIM:

• switch other and base order in SSIM (both spatial and mean)
• switch order in recipes
• switch reference values to use base rather than other

[James Warner (jwarner8)]

TODO check for impacts on cset_comparison_base through code

[ukmo-huw-lewis]

For discussion - propose there may be value in a further re-design here to remove explicit reference to "BASE_MODEL" and "OTHER_MODEL" in the *_difference recipe. I wonder if some of this originated from difference plots pre-dating CSET functionality to work with multiple model inputs (e.g. for overplotting N models in timeseries, histograms etc).

At present, if looking to run surface_spatial_difference recipe for 2 models, I provide input variables as follows.....
cset bake -r ${path_to_recipes}/recsurface_fields/surface_spatial_difference.yaml --input-dir "${input_dir}" "${input_dir2}" --output-dir ${output_dir} ... --BASE_MODEL="${model_name}" --OTHER_MODEL="${model_name2}"
(i.e. the 2 input model paths are caught in 1 definition of input-dir but I separate out naming of BASE and OTHER.

For other recipes with multiple model inputs (e.g. time series), we now call with something like....
cset bake -r ${path_to_recipes}/surface_fields/generic_surface_domain_mean_time_series.yaml --input-dir "${input_dir}" "${input_dir2}" --output-dir ${output_dir} ... --MODEL_NAME="['${model_name}', '${model_name2}']"
(i.e. MODEL_NAME is a list of inputs).

I would propose adapting difference recipes to take a single MODEL_NAME input variable consistent with other recipes, and then adopt protocol of all differences as model[N] - model[0] in the list. This means e.g. analysis can be defined as first model in list and will always be identified as the 'control'.

Argument against this is that spatial_difference recipe only expects 2 model inputs (i.e. does not actually loop over N models). Is the ability to do this (remove looping over BASE/OTHER combinations in workflow) desirable? For command-line use, I could envisage benefit for users to "give me differences of my N models relative to 0" for a list of models > 2.

Happy if this gets pushed to separate issue, but hope useful to flag here if this PR starts to touch on relevant parts of differencing infrastructure.

[James Frost (jfrost-mo)]

OFFICIAL

I would propose adapting difference recipes to take a single MODEL_NAME input variable consistent with other recipes, and then adopt protocol of all differences as model[N] - model[0] in the list. This means e.g. analysis can be defined as first model in list and will always be identified as the 'control'.

This would be a sensible change to reduce the unique behaviour of these recipes. It should probably be a separate pull request, both to keep this one simple, and as it will have lots to clean up (~~such as I think the cset_base_model attribute will be unnecessary~~) - No, it will still be needed.

For command-line use, I could envisage benefit for users to "give me differences of my N models relative to 0" for a list of models > 2.

Currently the loader in the workflow is what loops over all the models and fires them off, but each generated recipes only handles a single pair of models. Given the correspondence of one recipe to one diagnostic, I can’t see how we would handle multiple model differences in a single recipes (outside of line plots of bias with lots of models on, is that what you want?). OFFICIAL

[James Frost (jfrost-mo)]

Anke F (@afinnen) this is the reordering of the difference plots pull request.

[James Frost (jfrost-mo)]

#2061 has been opened to track Huw's comment.

[James Warner (jwarner8)]

Can I check the latest on this branch? It looks like SSIM has been completed. Frost opened an issue to track Huws comment. Is this ready for review?

[James Frost (jfrost-mo)]

With SSIM completed I'm happy to review this.

[James Frost (jfrost-mo)]

Looks good. We'll need to make sure we prominently include this in the release notes of our next release, but the updated recipes should help with that.

Grepping through the recipes I can't spot any missed reversals of the difference order, and the actual code change is simple enough.