Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add some tests to the test suites Chris Fischer runs #276

Closed
1 of 4 tasks
billsacks opened this issue Feb 13, 2018 · 5 comments · Fixed by #883
Closed
1 of 4 tasks

Add some tests to the test suites Chris Fischer runs #276

billsacks opened this issue Feb 13, 2018 · 5 comments · Fixed by #883
Assignees
Labels
priority: high High priority to fix/merge soon, e.g., because it is a problem in important configurations type: tests additions or changes to tests
Milestone

Comments

@billsacks
Copy link
Member

billsacks commented Feb 13, 2018

In a discussion with @mvertens , @jedwards4b , @fischer-ncar @cacraigucar, we decided that Chris will not regularly run the aux_clm test suite. Instead, we will add some tests to the prealpha and prebeta test suites if we feel there are additional tests that could catch cime issues that currently slip through the cracks.

In addition, we decided that there should be a new test category, aux_cime_baselines. This will likely be a subset of the prealpha test suite. This should be a small test suite (ideally no more than 3 or 4 tests defined by each component) whose purpose is to make sure that changes in cime have not changed answers relative to baselines for important configurations - i.e., that no answer changes have accidentally crept into cime. We want to keep this test suite small (even if that means not quite complete coverage) because it will be run frequently. It should be sufficient to have a few short SMS tests on a single machine/compiler (cheyenne_intel) - maybe covering a few different time periods and/or datm modes?

  • Review prealpha tests defined by ctsm for completeness. @ekluzek have you already done this recently? Also, make sure that all prealpha tests are also in aux_clm (or have very similar tests in aux_clm).

  • Review prebeta tests defined by ctsm for completeness. @ekluzek have you already done this recently? Also, make sure that all prebeta tests are also in aux_clm (or have very similar tests in aux_clm).

  • Define a new set of tests in ctsm's testlist xml file - aux_cime_baselines - as per the above notes.

  • Make sure that all aux_cime_baselines tests have similar tests in prealpha (see Add some tests to the test suites Chris Fischer runs #276 (comment))

cc @ekluzek @bandre-ucar

@billsacks billsacks added type: enhancement new capability or improved behavior of existing capability priority: high High priority to fix/merge soon, e.g., because it is a problem in important configurations labels Feb 13, 2018
@billsacks billsacks added this to To do in Rework test list Feb 13, 2018
@billsacks billsacks added type: tests additions or changes to tests and removed type: enhancement new capability or improved behavior of existing capability labels Jun 14, 2018
@billsacks
Copy link
Member Author

@ekluzek I'm assigning this to you. I think you've done some of it, but some items may need to be reviewed.

@billsacks
Copy link
Member Author

@ekluzek - I noticed that some of the aux_cime_baselines tests do not have similar counterparts in the prealpha list. I feel they should, for reasons that I note under aux_cime_baselines here: https://github.com/ESCOMP/ctsm/wiki/System-Testing-Guide#test-categories (@ekluzek - I just added that section to the wiki; feel free to edit as you see fit). So I think the aux_cime_baselines list and/or the prealpha list should be tweaked a bit so that all tests in aux_cime_baselines have similar counterparts in prealpha.

Specific tests that I see in aux_cime_baselines for which I can't see similar tests in prealpha are:

ERP_D_Ld10_P36x2.f10_f10_musgs.IHistClm50BgcCrop.cheyenne_intel.clm-ciso_decStart
ERP_P36x2_D_Ld5.f10_f10_musgs.I2000Clm45Sp.cheyenne_intel.clm-default
SMS_D_Ld3.f10_f10_musgs.I1850Clm50BgcCrop.cheyenne_intel.clm-default

I'm adding a checklist item to this effect in the original, top-level comment.

@ekluzek
Copy link
Contributor

ekluzek commented Aug 13, 2019

OK, I've gone through this and am bringing this in on the release-branch. prealpha/prebeta tests also have a aux_clm test, and aux_cime_baselines have a closely matched prealpha test.

@ekluzek
Copy link
Contributor

ekluzek commented Aug 13, 2019

Added prebeta tests for: SSP, Fates, drydep, Qian, fire-emis, cam6lndTuning, ciso, BgcSpinup, PET, DAE, waccm-deck 1&2 degree, and some more gnu on hobart and cheyenne. Add SSP, PEM, BgcSpinup, SMS_D_Ld3.f10_f10_musgs.I1850Clm50BgcCrop, and ERP_P180x2_D_Ld5.f19_g17.I2000Clm50Sp for prealpha.

@billsacks
Copy link
Member Author

See also #800

High priority automation moved this from Done but not on master to Done May 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority: high High priority to fix/merge soon, e.g., because it is a problem in important configurations type: tests additions or changes to tests
Projects
Development

Successfully merging a pull request may close this issue.

2 participants