Add suite keyword to suite definition file?

[gold2718]

In order to control the complexity of suites, we could allow a <suite> keyword in the suite definition file (SDF) in order to allow for some modularity. For instance, if a suite needs to run a chemistry suite in the middle of a group, it could just add that suite in the desired place. Moving chemistry then becomes as simple as moving a single line in the SDF.
The framework would treat a <suite> keyword much like a C pre-processor #include statement -- it would simply (recursively) add the entries of the referenced SDF as if they were part of the main SDF being processed.
Post-framework displays of the suite would act as if the suite was monolithic (no indication that there were sub-SDFs) or could include a comment to mark those locations.
Example with two SDFs

<suite name="B" version="1.0">
   <group name="G">
      <scheme>A</scheme>
      <scheme>S</scheme>
   </group>
</suite>

<suite name="A" version="1.0">
   <group name="phys">
      <scheme>B</scheme>
      <scheme>L</scheme>
      <suite name="B" group="G" />
      <scheme>T</scheme>
   </group>
</suite>

would be equivalent to:

<suite name="A" version="1.0">
   <group name="phys">
      <scheme>B</scheme>
      <scheme>L</scheme>
      <scheme>A</scheme>
      <scheme>S</scheme>
      <scheme>T</scheme>
   </group>
</suite>

Another use would be to group suites together but not merge them. In this case, we would have something like:

<suite name="physics" version="1.0">
   <group name="phys">
      <scheme>B</scheme>
      <scheme>L</scheme>
      <scheme>T</scheme>
   </group>
</suite>

<suite name="dycore" version="1.0">
   <group name="fast">
      <scheme>H</scheme>
      <scheme>R</scheme>
   </group>
</suite>

<suite name="UFS" version="1.0" merge="FALSE">
      <suite name="physics" />
      <suite name="dycore" />
</suite>

In this case, two suite caps would be generated which could be called from different parts of the executables acting as separate host models. If we adopt this approach, the first example above might need a merge="TRUE" attribute.

Thoughts?

[climbfuji]

We should talk about this in one of our following CCPP framework meetings, I'll put it on the agenda for the next one, knowing that it might be postponed because we have the NRL folks at the meeting.

[mkavulich]

We discussed this at today's CCPP Framework meeting and decided we do still want this feature.

[climbfuji]

@gold2718 I will pick this one up. Do we still want both options (merge true and false), or are we happy with just the merge = true option? I am in slight favor of implementing only the merge = true option, the need for the false option hasn't come up so far (afaik).

[gold2718]

@climbfuji, thanks for picking this up. I thought the merge=False version was for something like UFS where you have a physics suite and a "fast physics" suite to be executed in the dycore. merge=False would allow having both suites in one suite.xml file.
I do not think NCAR has a use for merge=False.

[climbfuji]

Thanks for the quick response. There was a lot of talking in the past of pushing physics into the dycore for a tighter integration with the dynamics on smaller time steps. But there has only ever been one physics scheme (GFDL MP) that was run inside the dycore, and that scheme is being replaced by other microphysics that don't have a "fast physics" (another term for the physics inside the dycore) part.

If the move towards more physics in the dycore ever happens, then I imagine that developers would like to split the suite definition files into two files to make them shorter.

[climbfuji]

@gold2718 I want to check in quickly with you. I started looking into this, progress is at glacial speed at best.

But at least I got to the point where I can ask a question or two!

Question 1

The example you are showing in the issue description has two suite blocks at the top level:

<suite name="B" version="1.0">
   <group name="G">
      <scheme>A</scheme>
      <scheme>S</scheme>
   </group>
</suite>

<suite name="A" version="1.0">
   <group name="phys">
      <scheme>B</scheme>
      <scheme>L</scheme>
      <suite name="B" group="G" />
      <scheme>T</scheme>
   </group>
</suite>

This is not valid in XML. By definition, there can be one and only one root element. I see two paths forward:

1. We want to be able to have multiple suites in one XML file. In this case, we need to introduce another root element, like this:

<suites>
  <suite name="B">
    ...
  </suite>
  <suite name="A">
    ...
  </suite>
</suites>

I think this is fine to do, but it requires updating all suites and it requires updating the schema and the XML parser in capgen.

2. We allow only one suite per XML file. In this case, no changes are needed for the existing suites and the parser. But the logic to find suites that may be inserted in other suites might become more difficult (depending on question 2 below).

There is a third option, which is to combine the bad parts of both options and allow both 1 and 2. Personally, I think 2 is the better option as it also makes more sense for reusing suites components in different parent suites, and it keeps the XML files shorter.

Question 2

Your initial idea reuses the suite tag inside a suite to pull in another suite. I haven't tried implementing this, but I assume it can be done. And even if it can't be done because of conflicts between the xml elements, then we could simply name it nested_suite.

However, XML has some limited native support for merging XML elements: The older external entity references method, and the newer http://www.w3.org/2001/XInclude feature. See https://www.xml.com/pub/a/2002/07/31/xinclude.html for a quick comparison.

My hope is that at least one of these would work in our case. My first impression is that the older method works only if we keep the suites in separate files, whereas the newer method may work for both. Each method will require some work, and maybe I'll find out along the way that they don't support our use case. But there are benefits using officially supported tools/syntax.

Thoughts? Ideas?

[gold2718]

Question 1, option 4

• Use XSD versioning. When I wrote the XSD file, I called it v1_0 and it is labeled as version="1.0" in the top-level xml element. Version 2.0 could use suites as the top level and the parser would recognize and properly parse both versions.

[gold2718]

Question 2: I am also in favor of using standard tools if they do not warp the intended use cases too much. I do not have time to look into them right now.

[gold2718]

Thought: For the use case where a set of related schemes or a scheme with it's interstitials is documented as a suite in a separate file (where it is also useful as a test case), I now think that the nested_suite keyword is more intuitive because it explicitly points to a separately-existing suite that (optionally?) lives in a separate ile.

[climbfuji]

Thanks for sharing your thoughts and reminding me of the versioning. I'll do that. No need for you to spend time on this unless you are bored - happy to explore more and provide a few options next week.