Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Possibility To Run All Tests From Directory #670

sprankhub opened this issue Apr 1, 2020 · 6 comments

Possibility To Run All Tests From Directory #670

sprankhub opened this issue Apr 1, 2020 · 6 comments


Copy link

I would like to have the ability to run all MFTF tests from a specific directory.


  1. Magento 2.3.4
  2. MFTF 2.5.3

Steps to reproduce

  1. Put all your custom modules under one specific directory. This could just be app/code, although we put them under src.
  2. Try to find a way to run all MFTF tests from this directory without editing the tests themselves and without listing every module separately.

Expected result

  1. All MFTF tests from this directory can be executed with one simple command.

Actual result

  1. Executing all MFTF tests from one directory with one simple command seems to be impossible currently.

With unit or integration tests, I can simply define a phpunit.xml file and include all tests from specific directories by using wildcards etc. This seems to be impossible with MFTF tests. Possibilities I thought of:

  • Use Suites: Adding a Suite configuration in dev/tests/acceptance/tests/_suite similar to the suite.xml.sample one seems to be nearly what I want. However, I need to list all modules explicitly in there. This is bad, because new modules are then not automatically included in the test execution.
  • Use Groups: This has the big disadvantage that I need to edit every test in the directory and add the same group to it. Hence, this is not an option.
Copy link
Member Author

I tried this now and it turns out that the Suites approach does not even work! When creating a custom suite under dev/tests/acceptance/tests/_suite/ and including a module via <module name="Namespace_Module"/>, the SuiteObjectExtractor only looks for Tests under dev/tests/acceptance/tests/functional/Magento/FunctionalTest/Namespace_Module/Test/*.xml. However, the tests of course reside in the respective module's folder, so e.g. app/code/Namespace/Module/Test/Mftf/. Hence, no tests are found at all and the following error is shown:

In SuiteObjectExtractor.php line 84:
    Suites must not be empty. Suite: "SampleSuite"

Hence, using the Groups approach seems to be the only possible workaround.

@okolesnyk okolesnyk added this to TODO in Community Issues via automation Apr 4, 2020
Copy link

fooman commented Apr 7, 2020

fyi I use something like

            <group value="Fooman_All"/>
            <group value="Fooman_PdfCustomiser"/>

and then I can run the following suite

    <suite name="Fooman">
            <group name="Fooman_All" />

but this is with tests copied into the dev/tests hierarchy.

Copy link
Member Author

Thanks @fooman, but yes, this has two disavantages:

  1. Tests need to be copied to the dev/tests directory.
  2. All tests have to have a specific group. If I forget to add the group, tests will not be executed.

Copy link

jilu1 commented Jul 27, 2020

@sprankhub Thanks for sending this feature request! Internal ticket is created

Copy link

Hi @sprankhub
We will close the current issue and would like you to try the --filter option for running specific tests.
Here's a link with a short description of how it works

Please feel free to create another issue or open PR if you think that filter's functionality isn't covering your case.

Community Issues automation moved this from TODO to Done Mar 22, 2022
Copy link
Member Author

With the --filter option, I can only run by group, which has the big disadvantage that I need to remember adding the right group, so that the tests are executed. I already mentioned this in the initial issue description:

Use Groups: This has the big disadvantage that I need to edit every test in the directory and add the same group to it. Hence, this is not an option.

I really think there should be another option.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

No branches or pull requests

4 participants