Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RunFailed module is trying to run removed test #5938

Closed
fkupper opened this issue Jun 15, 2020 · 10 comments
Closed

RunFailed module is trying to run removed test #5938

fkupper opened this issue Jun 15, 2020 · 10 comments

Comments

@fkupper
Copy link
Contributor

fkupper commented Jun 15, 2020

What are you trying to achieve?

  • Create SomeUnitTest.php test file with at least one test that fails
  • Run the tests
  • Delete the created test file
  • Run tests again
  • RunFailed extension will try to run the deleted file and abort the tests

What do you get instead?

$ codecept run -vvv
Codeception PHP Testing Framework v4.1.6
Powered by PHPUnit 6.5.14 by Sebastian Bergmann and contributors.
Running with seed: 

[Seed] 1724191921

In GroupManager.php line 129:
                                                                                             
  [Codeception\Exception\ConfigurationException]                                             
  GroupManager: File or directory /home/vagrant/my-project/tests/unit/SomeUnitTest.php does not exist                                                        
                                                                                             

Exception trace:
  at /home/vagrant/my-project/vendor/codeception/codeception/src/Codeception/Lib/GroupManager.php:129
 Codeception\Lib\GroupManager->checkIfFileExists() at /home/vagrant/my-project/vendor/codeception/codeception/src/Codeception/Lib/GroupManager.php:118
 Codeception\Lib\GroupManager->normalizeFilePath() at /home/vagrant/my-project/vendor/codeception/codeception/src/Codeception/Lib/GroupManager.php:81
 Codeception\Lib\GroupManager->loadConfiguredGroupSettings() at /home/vagrant/my-project/vendor/codeception/codeception/src/Codeception/Lib/GroupManager.php:25
 Codeception\Lib\GroupManager->__construct() at /home/vagrant/my-project/vendor/codeception/codeception/src/Codeception/SuiteManager.php:65
 Codeception\SuiteManager->__construct() at /home/vagrant/my-project/vendor/codeception/codeception/src/Codeception/Codecept.php:191
 Codeception\Codecept->runSuite() at /home/vagrant/my-project/vendor/codeception/codeception/src/Codeception/Codecept.php:163
 Codeception\Codecept->run() at /home/vagrant/my-project/vendor/codeception/codeception/src/Codeception/Command/Run.php:503
 Codeception\Command\Run->runSuites() at /home/vagrant/my-project/vendor/codeception/codeception/src/Codeception/Command/Run.php:397
 Codeception\Command\Run->execute() at /home/vagrant/my-project/vendor/symfony/console/Command/Command.php:255
 Symfony\Component\Console\Command\Command->run() at /home/vagrant/my-project/vendor/symfony/console/Application.php:987
 Symfony\Component\Console\Application->doRunCommand() at /home/vagrant/my-project/vendor/symfony/console/Application.php:255
 Symfony\Component\Console\Application->doRun() at /home/vagrant/my-project/vendor/symfony/console/Application.php:148
 Symfony\Component\Console\Application->run() at /home/vagrant/my-project/vendor/codeception/codeception/src/Codeception/Application.php:117
 Codeception\Application->run() at /home/vagrant/my-project/vendor/codeception/codeception/app.php:46
 {closure}() at /home/vagrant/my-project/vendor/codeception/codeception/app.php:47
 require() at /home/vagrant/my-project/vendor/codeception/codeception/codecept:7

run [-o|--override OVERRIDE] [-e|--ext EXT] [--report] [--html [HTML]] [--xml [XML]] [--phpunit-xml [PHPUNIT-XML]] [--tap [TAP]] [--json [JSON]] [--colors] [--no-colors] [--silent] [--steps] [-d|--debug] [--bootstrap [BOOTSTRAP]] [--no-redirect] [--coverage [COVERAGE]] [--coverage-html [COVERAGE-HTML]] [--coverage-xml [COVERAGE-XML]] [--coverage-text [COVERAGE-TEXT]] [--coverage-crap4j [COVERAGE-CRAP4J]] [--coverage-phpunit [COVERAGE-PHPUNIT]] [--no-exit] [-g|--group GROUP] [-s|--skip SKIP] [-x|--skip-group SKIP-GROUP] [--env ENV] [-f|--fail-fast] [--no-rebuild] [--seed SEED] [--no-artifacts] [--] [<suite>] [<test>]

Provide test source code if related

// paste test

Details

  • Codeception version: 4.1.6
  • PHP Version: 7.3
  • Operating System: ubuntu
  • Installation type: Composer
@fkupper fkupper changed the title RunFailed module is trying to run removed file RunFailed module is trying to run removed test Jun 15, 2020
@Naktibalda
Copy link
Member

Naktibalda commented Jun 20, 2020

I agree that this error is annoying when it happens, but this behaviour isn't limited to RunFailed and in other circumstances users may want to know that the test listed in the group does not exist.

It was implemented by #5792 as a response to comments in #5735

Personally I'd rather know I've misconfigured something so that I can fix it.

Can confirm it was a bad path in my codeception.yml group. 🤦‍♂️
I'd probably prefer an explicit exception be thrown in that case as well.

@fkupper
Copy link
Contributor Author

fkupper commented Jun 20, 2020

I agree that a specific error message or exception would also solve the issue. If it is explicited that the test that is missing is being loaded from RunFailed, it should be fine.

@Naktibalda
Copy link
Member

Naktibalda commented Jun 20, 2020

Oh, I didn't know that all groups are verified before run, I thought that it only affects codecept run -g failed.

@fkupper
Copy link
Contributor Author

fkupper commented Jun 20, 2020

Sorry that I did not described it explicitly enough. I thought that by not mentioning the use of the failed parameter it would be implicit that I was not using it in my case.

@chylex
Copy link

chylex commented Sep 19, 2020

This was very confusing behavior when I started using Codeception, and still continues to trip me up.

It's not clear at all that simply running codecept run, and then crashing with a GroupManager error complaining about missing files, means I have to go into /tests/_output and delete failed. There's no indication that GroupManager is trying to re-run failed tests on its own, it doesn't tell me which group is supposed to have the file, so everytime it happens I go hunting through .suite.yml files before I realize it's the failed tests again...

@Naktibalda
Copy link
Member

Naktibalda commented Jan 2, 2021

Would it be better if error message was GroupManager: File or directory /home/vagrant/my-project/tests/unit/SomeUnitTest.php set in failed group does not exist ?

@Naktibalda
Copy link
Member

Naktibalda commented Jan 2, 2021

Or

GroupManager: File or directory /home/vagrant/my-project/tests/unit/SomeUnitTest.php set in group failed does not exist

@chylex
Copy link

chylex commented Jan 2, 2021

Specifying the group in error message would help, but since the RunFailed extension is creating and running a new group of tests on its own, I would personally find it more helpful if it also had special treatment for missing tests - for example detecting this situation, and asking the user whether they want to proceed anyway, or halt and resolve the issue manually.

@Naktibalda
Copy link
Member

Naktibalda commented Jan 3, 2021

but since the RunFailed extension is creating and running a new group of tests on its own

RunFailed extension doesn't do any running, it only updates the group file.
Validation of failed file to RunFailed extension would probably triple the size of extension.

@lewiscowleschipuk
Copy link

lewiscowleschipuk commented May 4, 2022

This is what worked for me. YMMV

docker exec -it {container} bash -c 'rm -rf tests/_output/*'

This is from #5938 (comment) but in docker form as it took me a while to understand this was the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants