Passing filtering tags `--tag` and `--name` through an environment variable

[laraPPr]

In the EESSI/software layer we use reframe to run a number of tests after software is build. We are trying to pass our filtering parameters such as --tag and -n through an environment variable $REFRAME_ARGS . Now when we put the the put these filtering parameters in an environment variable we generate the following command: reframe --tag=CI --tag=1_4_node --nocolor -n EESSI_OSU -n EESSI_LAMMPS --list. But the reframe command that is generated looks like this "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/ReFrame/4.6.2/bin/reframe --tag=CI --tag=1_4_node --nocolor '-n EESSI_OSU -n EESSI_LAMMPS' --list -v" and so the filtering breaks because -n EESSI_OSU -n EESSI_LAMMPS is put between ''. I'm not sure how this happens.

You can find the code which generates the command here https://github.com/casparvl/software-layer/blob/ebe999ff58e375b7b4c6e920238508f5ca9416b7/test_suite.sh#L215-L267

and here is part of our logs to indicate what happens:

Loaded mappings from 'tests/eessi_test_mapping/software_to_tests.yml'

Read software names from 'module_files.list.txt'

Software: pmt/1.2.0-GCCcore-12.3.0-CUDA-12.1.1 -> Tests: ['EESSI_OSU', 'EESSI_LAMMPS']

Adding default set of tests: ['EESSI_OSU', 'EESSI_LAMMPS']

Full list of tests to run: ['EESSI_OSU', 'EESSI_LAMMPS']

Argument string: -n EESSI_OSU -n EESSI_LAMMPS

^[[32mSuccesfully extracted names of tests to run: -n EESSI_OSU -n EESSI_LAMMPS^[[0m

Listing tests: reframe --tag=CI --tag=1_4_node  --nocolor -n EESSI_OSU -n EESSI_LAMMPS --list

... 

[ReFrame Setup]

  version:           4.6.2

  command:           "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/ReFrame/4.6.2/bin/reframe --tag=CI --tag=1_4_node --nocolor '-n EESSI_OSU -n EESSI_LAMMPS' --list -v"

...

Loaded 1045 test(s)

Generated 104 test case(s)

Filtering test cases(s) by name: 0 remaining

Filtering test cases(s) by tags: 0 remaining

Filtering test cases(s) by other attributes: 0 remaining

Final number of test cases: 0

[List of matched checks]

Found 0 check(s)

[vkarak]

Implementation-wise is straightforward to add this, but it would have to be done for all filtering options, so it's going to be at least 4-5 new environment variables.

Regarding the problem you're seeing looks like a shell quoting problem of the -n options. Where are these defined?

[laraPPr]

We have resolved for our EESSI bot. @casparvl don't really know what was causing it but it is no longer an issue for us.

[vkarak]

I am closing this then.