Possibility To Run All Tests From Directory

[sprankhub]

I would like to have the ability to run all MFTF tests from a specific directory.

Preconditions

1. Magento 2.3.4
2. MFTF 2.5.3

Steps to reproduce

1. Put all your custom modules under one specific directory. This could just be app/code, although we put them under src.
2. Try to find a way to run all MFTF tests from this directory without editing the tests themselves and without listing every module separately.

Expected result

1. All MFTF tests from this directory can be executed with one simple command.

Actual result

1. Executing all MFTF tests from one directory with one simple command seems to be impossible currently.

With unit or integration tests, I can simply define a phpunit.xml file and include all tests from specific directories by using wildcards etc. This seems to be impossible with MFTF tests. Possibilities I thought of:

• Use Suites: Adding a Suite configuration in dev/tests/acceptance/tests/_suite similar to the suite.xml.sample one seems to be nearly what I want. However, I need to list all modules explicitly in there. This is bad, because new modules are then not automatically included in the test execution.
• Use Groups: This has the big disadvantage that I need to edit every test in the directory and add the same group to it. Hence, this is not an option.

[sprankhub]

I tried this now and it turns out that the Suites approach does not even work! When creating a custom suite under dev/tests/acceptance/tests/_suite/ and including a module via <module name="Namespace_Module"/>, the SuiteObjectExtractor only looks for Tests under dev/tests/acceptance/tests/functional/Magento/FunctionalTest/Namespace_Module/Test/*.xml. However, the tests of course reside in the respective module's folder, so e.g. app/code/Namespace/Module/Test/Mftf/. Hence, no tests are found at all and the following error is shown:

In SuiteObjectExtractor.php line 84:
    Suites must not be empty. Suite: "SampleSuite"

Hence, using the Groups approach seems to be the only possible workaround.

fyi I use something like

        <annotations>
            <group value="Fooman_All"/>
            <group value="Fooman_PdfCustomiser"/>
        </annotations>

and then I can run the following suite

    <suite name="Fooman">
        <include>
            <group name="Fooman_All" />
        </include>
    </suite>

but this is with tests copied into the dev/tests hierarchy.

[sprankhub]

Thanks @fooman, but yes, this has two disavantages:

1. Tests need to be copied to the dev/tests directory.
2. All tests have to have a specific group. If I forget to add the group, tests will not be executed.

[jilu1]

@sprankhub Thanks for sending this feature request! Internal ticket https://jira.corp.magento.com/browse/MQE-2236 is created

[okolesnyk]

Hi @sprankhub
We will close the current issue and would like you to try the --filter option for running specific tests.
Here's a link with a short description of how it works https://devdocs.magento.com/mftf/docs/commands/mftf.html#generatetests

Please feel free to create another issue or open PR if you think that filter's functionality isn't covering your case.

[sprankhub]

With the --filter option, I can only run by group, which has the big disadvantage that I need to remember adding the right group, so that the tests are executed. I already mentioned this in the initial issue description:

Use Groups: This has the big disadvantage that I need to edit every test in the directory and add the same group to it. Hence, this is not an option.

I really think there should be another option.