Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setup framework for retrying failed Selenium tests. #3993

Merged
merged 1 commit into from Apr 29, 2017

Conversation

Projects
None yet
3 participants
@jmchilton
Copy link
Member

commented Apr 28, 2017

  • By default this won't occur locally, but you can set GALAXY_TEST_SELENIUM_RETRIES to a non-zero number to enable auto retrying tests that many times.
  • Capture the stack trace in the Selenium test error report directory - this will be useful for debugging problems that may fail once but pass on a subsequence attempt. Jenkins now captures these directories and includes their content in the test reports.
  • Document the Selenium test error report directory in run_tests.sh as well as this new retry variable.
  • Update the Jenkins test script to set this new variable to 1 so transient failures break the build much less frequently.
Setup framework for retrying failed Selenium tests.
- By default this won't occur locally, but you can set GALAXY_TEST_SELENIUM_RETRIES to a non-zero number to enable auto retrying tests that many times.
- Capture the stack trace in the Selenium test error report directory - this will be useful for debugging problems that may fail once but pass on a subsequence attempt. Jenkins now captures these directories and includes their content in the test reports.
- Document the Selenium test error report directory in run_tests.sh as well as this new retry variable.
- Update the Jenkins test script to set this new variable to 1 so transient failures break the build much less frequently.
@martenson

This comment has been minimized.

Copy link
Member

commented Apr 29, 2017

excellent!

@martenson martenson merged commit c93afce into galaxyproject:dev Apr 29, 2017

5 checks passed

api test Build finished. 275 tests run, 0 skipped, 0 failed.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
framework test Build finished. 148 tests run, 0 skipped, 0 failed.
Details
integration test Build finished. 34 tests run, 0 skipped, 0 failed.
Details
toolshed test Build finished. 579 tests run, 0 skipped, 0 failed.
Details
@jmchilton

This comment has been minimized.

Copy link
Member Author

commented May 1, 2017

Thanks for the merge @martenson.

It works - this Jenkins job https://jenkins.galaxyproject.org/job/selenium/172/ would have failed without this but it was retried and the job succeeded on the second attempt. We still have a screenshot and stacktrace for the failure though - it looks like one that would be fixed with #3992.

@jmchilton jmchilton deleted the jmchilton:selenium_fixes_10 branch May 1, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.