New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WebDriver] Improve stability of Codeception when Selenium Server is gone #3531

Closed
eXorus opened this Issue Sep 20, 2016 · 6 comments

Comments

Projects
None yet
2 participants
@eXorus
Contributor

eXorus commented Sep 20, 2016

What are you trying to achieve?

I'm trying to improve the stability of Codeception when the Selenium Server is gone or the Selenium Hub lost the connection with the node.

I would like to always have WebDriver Exception catched by Codeception and continue to the next test like we already have when PHP Exception is throw.

1) WelcomeCest: Try to test3
 Test  tests/acceptance/WelcomeCest.php:tryToTest3

  [RuntimeException] Call to undefined method AcceptanceTester::methodXXX

What do you get instead?

Some exception are throws and the shell is returned without information. So build is failed in jenkins, no failed file generated, no test result updated when xml is asked and the others tests are not runned.

[Facebook\WebDriver\Exception\WebDriverCurlException]                                      
  Curl error thrown for http GET to /session/3d3f9b91-49da-4523-b3bb-f939562b5f58/log/types  
  Failed to connect to localhost port 4446: Connection refused

[Facebook\WebDriver\Exception\UnknownServerException]                                            
  Error forwarding the new session Error forwarding the request 172.17.0.4:5555 failed to respond

[Facebook\WebDriver\Exception\UnknownServerException]                                           
  Session [f5dd0664-2c1b-4d20-b88e-e6d6c013b040] was terminated due to FORWARDING_TO_NODE_FAILED

Provide console output if related. Use -vvv mode for more details.

➜  ./vendor/bin/codecept run acceptance -vvv
Codeception PHP Testing Framework v2.2.4
Powered by PHPUnit 5.4.8 by Sebastian Bergmann and contributors.

Acceptance Tests (4) 
Modules: WebDriver, \Helper\Acceptance

WelcomeCest: Try to test
Signature: WelcomeCest:tryToTest
Test: tests/acceptance/WelcomeCest.php:tryToTest
Scenario --
 I am on url "http://codeception.com"
  [Host] http://codeception.com
 I wait 5
 I see "Elegant and Efficient Testing for PHP"
 I wait 5
 I click "Quick Start"
 I wait 5
 PASSED 

WelcomeCest: Try to test2
Signature: WelcomeCest:tryToTest2
Test: tests/acceptance/WelcomeCest.php:tryToTest2
Scenario --
 I am on url "http://codeception.com"
  [Host] http://codeception.com
 I wait 5
 I see "Elegant and Efficient Testing for PHP"


  [Facebook\WebDriver\Exception\WebDriverCurlException]                                      
  Curl error thrown for http GET to /session/8530c9a6-2a14-4501-81ab-f32c1ba163d0/log/types  
  Failed to connect to localhost port 4446: Connection refused                               


Exception trace:
 () at /home/hello/Code/vendor/facebook/webdriver/lib/Exception/WebDriverException.php:92
 Facebook\WebDriver\Exception\WebDriverException::throwException() at /home/hello/Code/vendor/facebook/webdriver/lib/Remote/HttpCommandExecutor.php:320
 Facebook\WebDriver\Remote\HttpCommandExecutor->execute() at /home/hello/Code/vendor/facebook/webdriver/lib/Remote/RemoteWebDriver.php:547
 Facebook\WebDriver\Remote\RemoteWebDriver->execute() at /home/hello/Code/vendor/facebook/webdriver/lib/Remote/RemoteExecuteMethod.php:37
 Facebook\WebDriver\Remote\RemoteExecuteMethod->execute() at /home/hello/Code/vendor/facebook/webdriver/lib/WebDriverOptions.php:189
 Facebook\WebDriver\WebDriverOptions->getAvailableLogTypes() at /home/hello/Code/vendor/codeception/codeception/src/Codeception/Module/WebDriver.php:389
 Codeception\Module\WebDriver->debugWebDriverLogs() at /home/hello/Code/vendor/codeception/codeception/src/Codeception/Module/WebDriver.php:374
 Codeception\Module\WebDriver->_failed() at /home/hello/Code/vendor/codeception/codeception/src/Codeception/Subscriber/Module.php:78
 Codeception\Subscriber\Module->failed() at n/a:n/a
 call_user_func() at /home/hello/Code/vendor/symfony/event-dispatcher/EventDispatcher.php:174
 Symfony\Component\EventDispatcher\EventDispatcher->doDispatch() at /home/hello/Code/vendor/symfony/event-dispatcher/EventDispatcher.php:43
 Symfony\Component\EventDispatcher\EventDispatcher->dispatch() at /home/hello/Code/vendor/codeception/codeception/src/Codeception/PHPUnit/Listener.php:124
 Codeception\PHPUnit\Listener->fire() at /home/hello/Code/vendor/codeception/codeception/src/Codeception/PHPUnit/Listener.php:50
 Codeception\PHPUnit\Listener->addError() at /home/hello/Code/vendor/phpunit/phpunit/src/Framework/TestResult.php:260
 PHPUnit_Framework_TestResult->addError() at /home/hello/Code/vendor/codeception/codeception/src/Codeception/Test/Feature/ErrorLogger.php:20
 Codeception\Test\Test->errorLoggerEnd() at /home/hello/Code/vendor/codeception/codeception/src/Codeception/Test/Test.php:100
 Codeception\Test\Test->run() at /home/hello/Code/vendor/phpunit/phpunit/src/Framework/TestSuite.php:753
 PHPUnit_Framework_TestSuite->run() at /home/hello/Code/vendor/codeception/codeception/src/Codeception/PHPUnit/Runner.php:98
 Codeception\PHPUnit\Runner->doEnhancedRun() at /home/hello/Code/vendor/codeception/codeception/src/Codeception/SuiteManager.php:162
 Codeception\SuiteManager->run() at /home/hello/Code/vendor/codeception/codeception/src/Codeception/Codecept.php:209
 Codeception\Codecept->runSuite() at /home/hello/Code/vendor/codeception/codeception/src/Codeception/Codecept.php:178
 Codeception\Codecept->run() at /home/hello/Code/vendor/codeception/codeception/src/Codeception/Command/Run.php:329
 Codeception\Command\Run->runSuites() at /home/hello/Code/vendor/codeception/codeception/src/Codeception/Command/Run.php:256
 Codeception\Command\Run->execute() at /home/hello/Code/vendor/symfony/console/Command/Command.php:256
 Symfony\Component\Console\Command\Command->run() at /home/hello/Code/vendor/symfony/console/Application.php:818
 Symfony\Component\Console\Application->doRunCommand() at /home/hello/Code/vendor/symfony/console/Application.php:186
 Symfony\Component\Console\Application->doRun() at /home/hello/Code/vendor/symfony/console/Application.php:117
 Symfony\Component\Console\Application->run() at /home/hello/Code/vendor/codeception/codeception/src/Codeception/Application.php:103
 Codeception\Application->run() at /home/hello/Code/vendor/codeception/codeception/codecept:33

run [--report] [--html [HTML]] [--xml [XML]] [--tap [TAP]] [--json [JSON]] [--colors] [--no-colors] [--silent] [--steps] [-d|--debug] [--coverage [COVERAGE]] [--coverage-html [COVERAGE-HTML]] [--coverage-xml [COVERAGE-XML]] [--coverage-text [COVERAGE-TEXT]] [--no-exit] [-g|--group GROUP] [-s|--skip SKIP] [-x|--skip-group SKIP-GROUP] [--env ENV] [-f|--fail-fast] [--no-rebuild] [--] [<suite>] [<test>]

To reproduct that, you only need to launch acceptance tests and stopped the selenium server during the test.

Details

  • Codeception version: 2.2.4
  • PHP Version: 5.6.11
  • Operating System: Linux (Ubuntu)
  • Installation type: Composer
  • List of installed packages (composer show) : fresh install
  • Suite configuration: default configuration for WebDriver
@Naktibalda

This comment has been minimized.

Show comment
Hide comment
@Naktibalda

Naktibalda Sep 20, 2016

Member

Are you going to implement that?

Member

Naktibalda commented Sep 20, 2016

Are you going to implement that?

@eXorus

This comment has been minimized.

Show comment
Hide comment
@eXorus

eXorus Sep 20, 2016

Contributor

No, I don't understand how the Exception became an error for some case and not for other.

If you give me some tips I can try.

Contributor

eXorus commented Sep 20, 2016

No, I don't understand how the Exception became an error for some case and not for other.

If you give me some tips I can try.

@Naktibalda

This comment has been minimized.

Show comment
Hide comment
@Naktibalda

Naktibalda Sep 20, 2016

Member

OK, I will try to reproduce it later.

Member

Naktibalda commented Sep 20, 2016

OK, I will try to reproduce it later.

@Naktibalda

This comment has been minimized.

Show comment
Hide comment
@Naktibalda

Naktibalda Sep 20, 2016

Member

Exception wasn't caught by PhpUnit because it happened outside test.
In your case it happened in _failed, but it can happen in _initialize and _before methods too.

A more graceful approach would be to markAsSkipped if it exception was caught in _initialize and suppress exception if it was caught in _failed.

Member

Naktibalda commented Sep 20, 2016

Exception wasn't caught by PhpUnit because it happened outside test.
In your case it happened in _failed, but it can happen in _initialize and _before methods too.

A more graceful approach would be to markAsSkipped if it exception was caught in _initialize and suppress exception if it was caught in _failed.

@eXorus

This comment has been minimized.

Show comment
Hide comment
@eXorus

eXorus Sep 21, 2016

Contributor

Nice it's work well.

Could you explain why in failed we need to suppress exception and why in initialize and before need to put it markAsSkipped ?

Contributor

eXorus commented Sep 21, 2016

Nice it's work well.

Could you explain why in failed we need to suppress exception and why in initialize and before need to put it markAsSkipped ?

@Naktibalda

This comment has been minimized.

Show comment
Hide comment
@Naktibalda

Naktibalda Sep 21, 2016

Member

If exception gets out of event handler method, it kills Codeception, so we have to catch it and do something with it.

_failed is called after the test run is complete, so the only thing we can do there, is to print something.
markTestSkipped could be used in _before so that Codeception does not try to run the test and fail in a different way.
_initialize is executed before suite and can be called by _before if restart parameter is set to true.
markTestSkipped does not work before suite, so the only thing that can be done in initialize is print the error.

Member

Naktibalda commented Sep 21, 2016

If exception gets out of event handler method, it kills Codeception, so we have to catch it and do something with it.

_failed is called after the test run is complete, so the only thing we can do there, is to print something.
markTestSkipped could be used in _before so that Codeception does not try to run the test and fail in a different way.
_initialize is executed before suite and can be called by _before if restart parameter is set to true.
markTestSkipped does not work before suite, so the only thing that can be done in initialize is print the error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment