Support tagging in FTR tests

[banderror]

Summary

Heya! TL;DR:

Is there a way to add multiple tags to an FTR test (on describe and/or it levels) and then be able to run tests filtered by tags you need in a certain situation?

A good example of that would be something like mocha-tags that allows to filter tests with AND and OR logic. Or something like that.

Existing FTR CLI parameters

Before filing this issue I tried to find an existing solution. I checked the docs on testing and the kbn-test package. Noticed there are these parameters:

--include-tag=tag  a tag to be included, pass multiple times for multiple tags
--exclude-tag=tag  a tag to be excluded, pass multiple times for multiple tags

Also noticed they are used to group tests on CI. Does it mean if I add a new tag, a new CI group will be created? I don't want to necessarily interfere with CI.

this.tags(['skipCloud', 'ciGroup2']);

Unfortunately I couldn't find documentation describing how this tagging exactly works, what's the logic (AND, OR), and how it's related to the CI.

Original issue

We at @elastic/security-detections-response team have some FTR tests in x-pack/.../detection_engine_api_integration. There are basic and security_and_spaces subfolders each containing (mostly) the same tests. The only difference is their config - one uses basic ES license while the other using trial.

Instead of maintaining two sets of duplicated tests, would be nice to be able to tag some tests as e.g. license-basic, some as license-trial, some with both. And then be able to do 2 test runs like that:

node scripts/functional_tests --config x-pack/test/detection_engine_api_integration/config-basic.ts --tags "is:license-basic"

node scripts/functional_tests --config x-pack/test/detection_engine_api_integration/config-trial.ts --tags "is:license-trial"

I tried to use the existing tags functionality in detection_engine_api_integration, but there's some issue with types:

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[banderror]

Might be related to:

• Kibana Developer Experience #70733
• [Automation] Improve testing experience with FTR #53117

[spalger]

Hey @banderror, the issue in your screenshot is that you're using an arrow function for your describe block, so the this context isn't set to the suite created by describe, (it's instead set by the parent context). If you switch to a function expression the types should work and the tags will be properly associated with your suite.

The flags are intended to function like the existing filtering flags from Mocha, but powered by more machine-readable values rather than pattern matching suite names; --include-tag (mimicking --grep) will only include tests that match any of the supplied tags, --exclude-tag (mimicking --grep --invert) will exclude any test which matches any of the supplied tags. I'm not exactly clear how to describe that as and/or, but we apply both include and exclude tags, tests that run must satisfy both conditions (and), but only need to match a single include or exclude tag to qualify (or), so it's kind of both...

To only run the license-basic tests you would just pass --include-tag=license-basic and any test which doesn't have this tag would fail to match and be excluded. If you also had some tests which were not supposed to be run, you could add a tag to them like no-ci and then pass --include-tag=license-basic --exclude-tag=no-ci and only the tests tagged with license-basic that didn't also have the no-ci tag would run.

Not sure if that helps 😬 but if it does I can work up a version of that description for the CLI --help text.

[banderror]

Hi @spalger, thanks a lot for your clarification and sorry for a delayed reply from my side.

the issue in your screenshot is that you're using an arrow function for your describe block

Oh, my bad, absolutely! It's just so common to use arrows in these test blocks so I even didn't think about that.

I used an anonymous function, tagged my test suite with a tag, and it worked well:

export default function ({ getService }: FtrProviderContext) {
  const supertest = getService('supertest');
  const esArchiver = getService('esArchiver');

  describe('Generating signals from source indexes', function () {
    this.tags('license-trial');

node scripts/functional_tests --config x-pack/test/detection_engine_api_integration/security_and_spaces/config.ts --include-tag=license-trial

I can work up a version of that description for the CLI --help text

That would be great for sure. I missed some clarification regarding that in the help text and also in the docs.

I also noticed (if I'm not mistaken) that the logic of combining tags works like that. If we have node scripts/functional_tests --config <path> --include-tag=in1 --include-tag=in2 --exclude-tag=ex1 --exclude-tag=ex2, then they're gonna be combined as (in1 || in2) && !(ex1 || ex2).

Thank you! Please close the issue at will or let me know if you prefer me to close it.