Override test attributes from plan

[lukaszachy]

Use case is #1843 but while at it we can extend it for require/recommend as well - so, why not to make it generic enough?
Or isn't it worth the trouble? (I see the use case for renamed packages (e.g. dual package or collections)

How about this syntax (PLAN)?

discover:
  how: fmf
  url: http://example.com/foo.git
  set:
    duration: '*2'
    require+: [ 'foo']
    require-: ['bar']
    recommend: []
  edit:
    - key: require
      pattern: 'foo'
      replace: 'foo-ng'  

I'm not good at UX but the my goal will be to allow :

• extend attibute (+) or set if not exists
• remove attribute value (-), no operation if not there
• set new attribute values
• run re.sub on all values of selected attribute
• in the case of duration be able to 'multiply' current value

[psss]

The overall idea sounds reasonable and useful. Furthermore, another related use cases are coming (e.g. enable the avc check for all tests). Regarding implementation, a quick brainstorm question: Would it make sense to use the adjust syntax here as well?

discover:
  how: fmf
  url: http://example.com/foo.git
  adjust:
    - duration+: 1h
    - require+: ['foo']
      when: distro == rhel-9

The when condition is optional so if no context limitations are to be applied the syntax could look quite concise and consistent with what users already know. Plus the implementation would be quite simple, probably just one additional adjust() call.

[LecrisUT]

+1 for this UX, would also addreas #2340 if it was made generic, i.e. adjust() for altering the imported plan, plans.prepare, etc. to run in the context of the calling plan

[happz]

Looks good, using adjust concept makes sense. But, I would propose using a different keyword instead of adjust, because adjust has an effect of modifying the object that contains it, while in this case, the discovered tests would be modified. Maybe something like adjust-tests & "it's the adjust you know and love, but applies to tests discovered by the phase."

discover:
  how: fmf
  adjust-tests:
    check+:
      - name: avc

Yeah, that should work :)

[LecrisUT]

Oh indeed, it would be confusing if it refers to adjust plan or adjust test, especially since there would only be 2 spaces distinguishing them

[jchecahi]

This feature would be highly valuable for a use case we're trying to implement: reusing LLVM tests in the compat packages. LLVM compat packages are named with a version suffix, similar to python3.x packages. In order to reuse a test, we must be able to modify a test's require list so instead of requesting clang, clang-libs etc, it requests clang17, clang17-libs, etc.

Implementing this without the feature proposed here would be a challenge. We'd need to either duplicate all the code using a branched scheme (one branch per compat package version) and replace all the ocurrences of clang with clangXX (XX = llvm version), or introduce a series of adjust: conditions based on context variables(?) that would difficult the maintainability. LLVM is comprised of 7 components so either option would need to be implemented in each repo.

The approach proposed here aligns perfectly with the use case, and would easily allow us to reuse them without actual changes in the tests:

discover:
  how: fmf
  url: https://src.fedoraproject.org/tests/clang.git
  edit:
    - key: require
      pattern: 'clang'
      replace: 'clang17'
    - key: require
      pattern: 'llvm'
      replace: 'llvm17'
    # And so on with the rest of LLVM components...

[lukaszachy]

Since we go with 'adjust' syntax I'd like to ask for help with deciding the way how adjust can edit values. From the implementation PoV I'd say re.sub with check that if the final string is empty the value will be removed from the list.

Motivation for the 'drop if empty' is that if test has require: [make, patch, python3-test, python3-foo ...] it is easier to drop all python3-* require using regexp than enumeration all to the require-:[...

Problem is how to write that in adjust syntax. I'm thinking of another merge operator, probably ~ suffix.
However - should the pattern/repl strings be in single string (similar to sed-like "s/regexp/replacement/") OR it will be better to have explicit pattern/replace keys processed during the adjust?

Lets' say that the discovered test has require: [ foo, bar]. UC is to change 'foo' to 'foo2-ng' and drop 'bar'

The 'sed-like' could be - Downside in escaping '/' (or allowing custom char to separate?)

discover:
  how: fmf
  adjust-tests:
  - require~:
     - "foo/foo2-ng"
     - "bar/"

The explicit way - too chatty, isn't it?

discover:
  how: fmf
 adjust-tests:
   - require~:
     - pattern: foo
       replace: foo2-ng
     - pattern: bar
       replace: ""

Edit: This merge operator will land in fmf package, but this discussion has already the audience and necessary background.

[LecrisUT]

Oh, I just realized, don't we have a cleaner approach with graft operations? For discover we just need to specify the tree path to graft on top or below it. Then the adjust are specific to context adjusts and always require when

[lukaszachy]

Grafting as adjust-test? I don't think so - number of nodes stays the same. IMO the 'edit' is more suitable name, but that is something we can change.

[LecrisUT]

I think it would offer more flexibility:

• Edit the base of all tests (reverse-graft): have a root.fmf tree and graft the discover tree on top of that
  • If only the root is populated, no new leaves are produced
• Edit specific tests (forward-graft): after the discover tree is produced, graft my_test.fmf on top of that
  • If names are for leaf tests, than you can overwrite any variable defined by discover. (Similar to having a /root/my_test defined in a top-level main.fmf and then adding a /root/my_test/main.fmf file on top of that)

And these can be mixed together. Ignoring my bad naming intuition a UX would be:

execute:
  how: tmt
discover:
  how: fmf
  url: https://src.fedoraproject.org/tests/clang.git
  graft-on: /path/to/base/tree
  graft-from: /path/to/leaf/tree

Where /path/to/base/tree could have a main.fmf as

environment:
  PACKAGE_TO_TEST: my_package

And /path/to/leaf/tree could have:

/root/specific_test:
  result: xfail

This format can be better for the user because:

• User can make arbitrary tree structures (new .fmf/version) to fit their need
• Effect of adjust is well-defined within the file

[lukaszachy]

IMO that is too complicated -- use case is to change some attributes as easy as possible.

[LecrisUT]

True, but I think it should be addressed at some point, particularly as we work on plugins/frameworks that dynamically generate tests. The adjust-tests does not cover adjusting and individual test, for which this approach could do. (probably adding when logic would complicate stuff more on both developer and user side)

[lukaszachy]

Work in progress - #1843 (comment)