Alter `steps_to_run` in config file or command line #104

xylar · 2021-04-30T21:26:58Z

This merge adds the ability for users to choose which steps in a test case they want to run. They have 2 options.

First, they can alter the steps_to_run config option in the test_case section at the very top of the test case's config file:

[test_case]
steps_to_run = initial_state full_run restart_run

The list of steps_to_run can be separated by spaces, commas or both.

Or they can specify which steps to run (or which steps not to run) via the --steps (or --no-steps) flag to compass run:

python -m compass run --steps initial_state full_run
python -m compass run --no-steps restart_run

Both of these accomplish the same thing -- skipping the restart_run step of the test case.

The new capability has been added to the documentation.

closes #80

xylar · 2021-04-30T21:31:00Z

Testing

I have run the nightly test suite on Ubuntu with gnu, and everything passed and is bit-for-bit with master from earlier today.

I also verified that I could modify the config option steps_to_run and only the desired steps ran. Similarly, I verified that I could both explicitly include steps with the --steps flag and exclude them with the --no-steps flag. I verified that these flags have no effect when calling compass run to run a suite or within a step work directory.

xylar · 2021-04-30T21:32:35Z

@mark-petersen requested something like this in #80. This is my suggestion.

@mark-petersen and @matthewhoffman, if you are willing to review, I would appreciate a little testing. And please let me know if you are happy with this solution or if you have alternative suggestions or changes.

mark-petersen · 2021-05-10T15:03:00Z

I tested this at a test case level and it works, both with flags and *.cfg file. Thanks, that is handy!

One of my common workflows is to run the full nightly regression suite once, then rerun it with other compilers or debug, openmp settings, but turn off the QU240 mesh and init to save time. Is there a way to do that? I think this PR does not address omitting a test case from a suite altogether.

I tried it with this functionality, ie

vi ocean/global_ocean/QU240/mesh/mesh.cfg
steps_to_run =

with a blank, but it ran that step anyway. The suite function must override the config setting.

xylar · 2021-05-10T16:44:08Z

@mark-petersen,

One of my common workflows is to run the full nightly regression suite once, then rerun it with other compilers or debug, openmp settings, but turn off the QU240 mesh and init to save time. Is there a way to do that? I think this PR does not address omitting a test case from a suite altogether.

My suggestion would be to make version of the nightly suite with and without the test you don't want to run. I can't think of any way to efficiently specify that you don't want to run a particular step but you could open an issue with a suggested method for doing that.

I tried it with this functionality, ie
...
with a blank, but it ran that step anyway. The suite function must override the config setting.

Hmm, I can look into that. If steps_to_run isn't getting processed during test-suite runs, that isn't intentional. I'll get back to you..

xylar · 2021-05-12T09:47:21Z

@mark-petersen, indeed, steps_to_run wasn't getting parsed for test cases run out of a test suite. I have fixed this now and run the nightly suite once with with mesh step and once without it, as you suggested. It worked for me. Could you verify that it works for you?

Please also comment on #114 so I can make sure to address your needs there.

matthewhoffman

@xylar , this looks great. I added inline comments for the 2 items we discussed on the phone.

matthewhoffman · 2021-05-25T16:15:41Z

docs/developers_guide/command_line.rst

+    python -m compass run --no-steps restart_run
+
+Would both accomplish the same thing in this example -- skipping the
+``restart_run`` step of the test case.


Add note that using command line option will override the cfg version.

matthewhoffman · 2021-05-25T16:16:56Z

compass/run.py

+                                  'steps_to_run').replace(',', ' ').split()
+
+    if steps_not_to_run is not None:
+        steps_to_run = [step for step in steps_to_run if step not in


Check if step name specified is actually a valid step name or not. (This should be done for both --steps and --no-steps.)

xylar · 2021-05-25T21:35:43Z

@matthewhoffman, thank you for both of your excellent suggestions. I believe they are now addressed. Please give this another look when you have a chance.

mark-petersen

Looks good. Thanks for these improvements.

xylar · 2021-06-04T07:08:49Z

@matthewhoffman, still waiting on a re-review here. Is that something you can do soon?

This will be at the top of the config file for every test case and will contain the default steps to run, which the user can then alter before running the test case.

These are used to explictly run (or not run) steps in the test case.

When a bad step name is given to the `steps_to_run` config option or the `--steps` or `--no-step` command-line flags, an error is raised.

matthewhoffman · 2021-06-04T16:19:09Z

I'm looking at this again now. I got myself confused because I hadnt realized it was rebased on the dev env.

matthewhoffman

Looks good @xylar. Thanks for adding the additional error-checking. I tried all the ways of asking it to do incorrect things, and it gave sensible errors each time.

xylar · 2021-06-04T17:38:22Z

Thanks!

xylar added enhancement New feature or request ocean labels Apr 30, 2021

xylar requested review from mark-petersen and matthewhoffman April 30, 2021 21:26

xylar self-assigned this Apr 30, 2021

xylar added this to In progress in compass 1.0 via automation Apr 30, 2021

xylar added python package DEPRECATED: PRs and Issues involving the python package (master branch) and removed ocean labels May 4, 2021

xylar mentioned this pull request May 12, 2021

Need a method to specify which tests of a test suite to run (or not run) #114

Closed

matthewhoffman requested changes May 25, 2021

View reviewed changes

mark-petersen approved these changes May 26, 2021

View reviewed changes

xylar added 6 commits June 4, 2021 09:09

Add steps_to_run config option

ebe9efc

This will be at the top of the config file for every test case and will contain the default steps to run, which the user can then alter before running the test case.

Add compass run flags --steps and --no-steps

f1d0032

These are used to explictly run (or not run) steps in the test case.

Add documentation of the new options

ae392ec

Read steps_to_run for suites, too

6c85971

Clarify that command-line flags take presedence

ead41b9

Raise errors when on incorrect step name

cd179a0

When a bad step name is given to the `steps_to_run` config option or the `--steps` or `--no-step` command-line flags, an error is raised.

xylar force-pushed the alter_steps_to_run branch from 03ccde2 to cd179a0 Compare June 4, 2021 07:10

matthewhoffman self-requested a review June 4, 2021 16:23

matthewhoffman approved these changes Jun 4, 2021

View reviewed changes

xylar merged commit 5ab0531 into MPAS-Dev:master Jun 4, 2021

compass 1.0 automation moved this from In progress to Done Jun 4, 2021

xylar deleted the alter_steps_to_run branch June 4, 2021 17:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Alter `steps_to_run` in config file or command line #104

Alter `steps_to_run` in config file or command line #104

xylar commented Apr 30, 2021

xylar commented Apr 30, 2021

xylar commented Apr 30, 2021

mark-petersen commented May 10, 2021

xylar commented May 10, 2021

xylar commented May 12, 2021

matthewhoffman left a comment

matthewhoffman May 25, 2021

matthewhoffman May 25, 2021

xylar commented May 25, 2021

mark-petersen left a comment

xylar commented Jun 4, 2021

matthewhoffman commented Jun 4, 2021

matthewhoffman left a comment

xylar commented Jun 4, 2021

Alter steps_to_run in config file or command line #104

Alter steps_to_run in config file or command line #104

Conversation

xylar commented Apr 30, 2021

xylar commented Apr 30, 2021

Testing

xylar commented Apr 30, 2021

mark-petersen commented May 10, 2021

xylar commented May 10, 2021

xylar commented May 12, 2021

matthewhoffman left a comment

Choose a reason for hiding this comment

matthewhoffman May 25, 2021

Choose a reason for hiding this comment

matthewhoffman May 25, 2021

Choose a reason for hiding this comment

xylar commented May 25, 2021

mark-petersen left a comment

Choose a reason for hiding this comment

xylar commented Jun 4, 2021

matthewhoffman commented Jun 4, 2021

matthewhoffman left a comment

Choose a reason for hiding this comment

xylar commented Jun 4, 2021

Alter `steps_to_run` in config file or command line #104

Alter `steps_to_run` in config file or command line #104