rose macro: suite level functionality

[arjclark]

We should be able to run rose macro across a whole suite and retrieve errors (a.la. rose edit) rather than one app at a time.

@benfitzpatrick - if you find the duplicate supersede it with this.

[arjclark]

@kaday - would this be something you'd be interested in doing?

[kaday]

Yes, I will assign it to myself.

[matthewrmshin]

When this issue is resolved, is it worth thinking about how to connect rose macro --validate with rose suite-run? Do we want to provide a shortcut option like this?

rose suite-run --validate
# which will be roughly equivalent to:
rose macro --validate && rose suite-run

[arjclark]

When this issue is resolved, is it worth thinking about how to connect rose macro --validate with rose suite-run? Do we want to provide a shortcut option like this?

@matthewrmshin - one to discuss - at present majority of our core suites would fail that test for one reason or another.

[hjoliver]

The suite-run suggestion, I think, comes from an email conversation between @matthewrmshin and me - so I'll chime in with my thoughts here: I'd like the option to have suite-run abort if any inputs are found to be inconsistent with metadata, because these represent potential errors and suite-run is the final point at which such checks can be done before beginning execution. If optional this could provide some additional safety for important suites without preventing developers from usefully making quick changes that are inconsistent with metadata. (I was thinking this was analogous to cylc validate --strict failing naked dummy tasks etc., although by my logic I suppose cylc run/restart should have the same --strict option).

[arjclark]

I'd like the option to have suite-run abort if any inputs are found to be inconsistent with metadata, because these represent potential errors and suite-run is the final point at which such checks can be done before beginning execution. If optional this could provide some additional safety for important suites without preventing developers from usefully making quick changes that are inconsistent with metadata.

@hjoliver - I like the idea of having that option too - its more a reservation about how people are using the metadata. A number of our suites have, for example, UM apps with multiple optional configs for different modes. The core config is often stripped down to the shared settings with the contents for each of the ways it could be run (e.g. short step, long step, recon, forecast etc.) in optional configs. All together, with whatever combination of configs will be run, the config would validate but as far as rose macro etc. see it it fails the metadata tests. Do you see similar usage at NIWA with your UM apps?

[benfitzpatrick]

Just had an upvote on this from a user here.

[hjoliver]

I somehow missed the question from @arjclark above. At NIWA our use of UM apps is still rather primitive - UMUI 8.4 jobs converted with rose bud, no decent UM metadata for the resulting apps (it wasn't available at 8.4) .... which severely discourages non-experts from any subsequent messing around with UM apps (including any use of optional configs). Thanks to the shared repositories at UM 10 though, we are finally able to use the UM properly under the new infrastructure.... so watch this space.

[arjclark]

@kaday - as discussed at today's meeting I'm grabbing this from you with a view to prioritising it in the very near future.

[arjclark]

Looks like this is going to be a somewhat non-trivial change - as the code currently stands its not nicely set up for running in more than one place (or even accepting arguments for running on more than one app) so it might take a while - need to make sure changing it doesn't break anything else.

[kaday]

I did say that ages ago! And completely agree.

[matthewrmshin]

Looking back at the discussion on rose suite-run... rose suite-run has a --no-strict option, which makes it to call cylc validate without the --strict option. Otherwise, it calls cylc validate --strict by default. Therefore, rose suite-run should validate Rose suite and application configurations by default, and only switch off validation if a --no-strict (or --no-validate) option is specified.

[arjclark]

@matthewrmshin - please see the commentry above about the setup of our operational suites and the way people develop their suites. cylc validate and rose macro validation are two potentially quite different things. cylc validate is good to run because it stops you launching a suite that crashes out halfway through a run due to an invalid construction. rose macro --validate, however, checks if an app is aligned with its metadata - something that is 1) non-essential and 2) we've said people can go against if they're developing/know better than wrong/incomplete central metadata. I've no idea either how we'd bring in metadata for the multiple branches used in some of our operational suites such that they wouldn't fail to launch.

[matthewrmshin]

I believe we can extend the CLI to allow switching on/off validation of different parts of the suite.

[oliver-sanders]

Closed by #1884, moved the rose suite-run portion of this issue to #1908.