Filter tests by tags

[FlyingSamson]

Checklist

• Checked Support page

Is your feature request related to a problem? Please describe.
I'm trying to execute different TEST_CASEs defined in a single file with different executionWrappers. Specifically I have one TEST_CASE tagged "[serial]" and another tagged "[parallel]" and I would like to call the first one directly (without using an executionWrapper) and the second one with MPI using

"executionWrapper": {
    "path": "<path to mpirun>",
    "args": [ "-n", "2", "${cmd}", "${argsFlat}" ]
}

Describe the solution you'd like
One idea that came to my mind was to have a top-level option like filter_tags_regex that similar to pattern applies this advancedExecutables-object only to matching tests. I think you proposed a similar thing already in this comment.

So effectively I would like to define in my settings.json:

"testMate.cpp.test.advancedExecutables": [
  {
    "name": "${filename}",
    "pattern": "${command:cmake.buildDirectory}/tests/src/*{test,Test,TEST}*",
    "filter_tags_regex": "serial"
    ...
  },
  {
    "name": "${filename}",
    "pattern": "${command:cmake.buildDirectory}/tests/src/*{test,Test,TEST}*",
    "filter_tags_regex": "parallel",
    "executionWrapper": {
      "path": "<path to mpirun>",
      "args": [ "-n", "2", "${cmd}", "${argsFlat}" ]
    }
    ...
  }
]

to execute tests tagged "serial" on a single process, while tests tagged "parallel" would be run with MPI on two processes.

Describe alternatives you've considered
One option would of course be to split the test in two files to be able to discern them using pattern. But there could also be parts of the code that should work both on a single as well as multiple processes so I would have to duplicate the tests for such parts in two files to have them executed once with one and once with multiple processes.

[matepek]

Hello,

The challenge of that the filter should work with all the frameworks.

Yeah, you could have two different executables. The common code/tests part could go to a .h file and could be included (note that the preprocessor's include directive actually includes the whole content of the given file at any line). With that you could avoid the code duplication.

What do you think?