Behavior of fail-fast flag on 4.1.23?

[PZ01]

What are you trying to achieve?

Running tests inside of a Gitlab Pipeline. Expected to have folder specified in command line to be run only.

What do you get instead?

All tests ran, causing issues with our Pipeline

Provide console output if related. Use -vvv mode for more details.

echo "Running tests in folder ${CODECEPTION_TEST_FOLDER}"
Running tests in folder tests/unit
$ vendor/bin/codecept run --no-interaction --xml $CI_JOB_NAME.xml --fail-fast $CODECEPTION_TEST_FOLDER || EXIT_CODE=$?
Codeception PHP Testing Framework v4.1.23
Powered by PHPUnit 9.5.10 by Sebastian Bergmann and contributors.
[Seed] 242220195
Functional Tests (328) -------------------------------------
✖ PerformTest: It_should_return_something_specific (0.09s)

I saw fail-fast has been changed in 4.1.23, I set pipeline to 4.1.22 and it's working. Seems like if a default is not provided, it considers the path as a value to the flag or something? As you can see, the message 'Running tests in folder XXX' is actually running everything. I picked up on this because we have around 100 unit tests, and I saw that codecept started running everything(328).

It was my bad to keep ^4.0 in the pipeline, but I didn't expect the problem to be of this nature. --fail-fast without a number shouldn't be an impediment for folder path provided at the end.

Details

• Codeception version: 4.1.23
• PHP Version: 7.4
• Operating System: Linux
• Installation type: Composer

[b1rdex]

That's a breaking change introduced in #6275. We've got the problem too.

[b1rdex]

@Verest do you have any ideas how to solve this? Is it possible to check whether the value is an int (new behavior) or a path?

[Verest]

I looked and the section responsible for setting the arguments is within Symfony\Component\Console\Input\ArgvInput i.e. the third party. This is called within src/Codeception/Command/Run.php::execute() via $input->getOptions().

Symfony's documents state that a -- should be added after if not an option, I wasn't aware of this until now unfortunately. See: https://symfony.com/doc/current/components/console/console_arguments.html

From a cursory look, it seems php codecept run cli --fail-fast tests/cli/RunCest.php has the issue replicated here. php codecept run cli --fail-fast -- tests/cli/RunCest.php does fix it.

In retrospect, this shoulda been merged into develop as it does seem to be a breaking change. That said, I do not think this is a bug, rather expected behavior based on the Symfony docs.

[Verest]

I am running into issues adding unit tests themselves to test this. The executeCommand helper method would need to be updated as it is appending a -n to the end which will break any filepaths added onto the $command passed in. See tests/supportCliHelper.php.

[Naktibalda]

I think that we have to revert fail-fast change in 4.1 branch.

[Naktibalda]

Reverted in 4.1.26

@Verest I merged your feature to 5.0 branch, which should be released in a few weeks.

You can keep using 4.1.25 if you really need this feature.