XML macros: add named yields, tokenized macros, tokens for attributes, and fix replacement of toplevel yield

[bernt-matthias]

Originally I wanted to find out what exactly happens if there are multiple yield tags.. hoping that I can somehow control that each yield is replaced with the corresponding child of the expand tag. This did not work. So I added named yields and two other extensions that I were missing in the past:

Named yields

Essentially <yield name="XYZ"/> is replaced by the content of <template name="XYZ"/>. Unnamed yields work as before and named and unnamed yields can be mixed.

An example:

    <xml name="named_yields_example">
      <conditional>
        <param type="select">
          <option value="a">A</option>
          <option value="b">B</option>
          <yield name="more_options"/>
        </param>
        <when value="a">
          <param type="select">
            <yield />
          </param>
        </when>
        <when value="b">
          <param ... />
        </when>
        <yield name="more_whens">
      </conditional>
    </xml>

        <expand macro="named_yields_example">
          <token name="more_options">
            <option value="c">C</option>
          </token>
          <token name="more_whens">
            <when value="c">
              <param type="select">
                <yield />
              </param>
            </when>
          </token>
          <options from_data_table="tophat2_indexes">
            <filter type="sort_by" column="2"/>
            <validator type="no_options" message="No genomes are available for the selected input dataset"/>
          </options>
        </expand>

The choice of token as tag name may be sub-optimal... since tokens are also used to parametrize macro parameters. Should be easy to change if desired.

Allow that the name of nested macros can be controlled by tokens:

For instance

<xml name="example">
    <expand macro="@TOKEN_NAME@"/>`
</xml>

See

galaxy/test/unit/tool_util/test_tool_loader.py

Line 611 in 372439c

def test_loader_specify_nested_macro_by_token():

for a complete example

Allow token expansion also for attributes

<tool>
    <macros>
        <token name="__ATTR_NAME">name</token>
    </macros>
    <another>
        <tag __ATTR_NAME="blah" />
    </another>
</tool>

Here the token can't be surrounded by @ character (i.e. @ATTR_NAME@) since the xml file needs to proper xml (because it is parsed before token/macro expansion). But it seems that for tokens surrounding the token by @ is a (undocumented) convention only, anyway.

Docs have been added here: galaxyproject/planemo#1212 (not all changes are yet documented, but will be when this is merged).

I'm happy if there are more suggestions to make this more expressive / powerful / elegant :). Real recursion would be cool but also dangerous .. also could not think of a way to express a stopping condition.

Fix: If there are multiple top level yield statements only the last was replaced

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

• I agree to license these contributions under Galaxy's current license.
• I agree to allow the Galaxy committers to license these and all my past contributions to the core galaxy codebase under the MIT license. If this condition is an issue, uncheck and just let us know why with an e-mail to galaxy-committers@lists.galaxyproject.org.

[mvdbeek]

We should run this against the iuc as well.

[mvdbeek]

deepcopy essentially pickles and unpickles the object, this could be detrimental to startup speed. Would a .copy() be sufficient ?

[bernt-matthias]

True. I remember that if I do not deepcopy then the following clear also removes the copied elements. Maybe calling remove(subelement) for each child helps.

In the end this is only a workaround: I was not able to find a way to iterate only the expand elements that are not included in the macros subtree. With lxml it might be possible, but it appears to me that the code needs to be compatible with xml from the stdlib and lxml, or am I wrong?

[mvdbeek]

You could import LXML_AVAILABLE and make that conditional on this (if it works).

[mvdbeek]

Are you sure this must be a cycle ? Also list lookups can be expensive, maybe make visited a set or an ordered set?

[bernt-matthias]

Are you sure this must be a cycle ?

Quite. Essentially this is a DFS of the expansions and if a previously expanded macro is expanded again we have a cycle. Note that visisted is kind of a stack (i.e. the macro names are removed at the end of the function.)

Also list lookups can be expensive, maybe make visited a set or an ordered set?

Then I will go for ordered set, since otherwise the test can not check for the message (or I just test for the prefix "Cycle in nested macros".

[mvdbeek]

This looks good, but let's have a look at the IUC tests in galaxyproject/tools-iuc#4332

[bernt-matthias]

Seems that the same 15 tools fail that also failed for #13039

[github-actions]

This PR was merged without a "kind/" label, please correct.