Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Matthew Robbins
committed
May 13, 2015
1 parent
b0d336a
commit 58de26a
Showing
4 changed files
with
153 additions
and
186 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1 @@ | ||
<a href="/tags#browsermob" title="Postings tagged browsermob" style="font-size: 10.5px; line-height:10.5px">browsermob</a> <a href="/tags#browsermob-proxy" title="Postings tagged browsermob-proxy" style="font-size: 10.5px; line-height:10.5px">browsermob-proxy</a> <a href="/tags#bundler" title="Postings tagged bundler" style="font-size: 11.0px; line-height:11.0px">bundler</a> <a href="/tags#capybara" title="Postings tagged capybara" style="font-size: 12.0px; line-height:12.0px">capybara</a> <a href="/tags#centos" title="Postings tagged centos" style="font-size: 10.5px; line-height:10.5px">centos</a> <a href="/tags#chrome" title="Postings tagged chrome" style="font-size: 10.5px; line-height:10.5px">chrome</a> <a href="/tags#ci" title="Postings tagged ci" style="font-size: 11.0px; line-height:11.0px">ci</a> <a href="/tags#cpaybara" title="Postings tagged cpaybara" style="font-size: 10.5px; line-height:10.5px">cpaybara</a> <a href="/tags#cucumber" title="Postings tagged cucumber" style="font-size: 13.0px; line-height:13.0px">cucumber</a> <a href="/tags#elixir" title="Postings tagged elixir" style="font-size: 10.5px; line-height:10.5px">elixir</a> <a href="/tags#erlang" title="Postings tagged erlang" style="font-size: 10.5px; line-height:10.5px">erlang</a> <a href="/tags#grid" title="Postings tagged grid" style="font-size: 10.5px; line-height:10.5px">grid</a> <a href="/tags#gvim" title="Postings tagged gvim" style="font-size: 10.5px; line-height:10.5px">gvim</a> <a href="/tags#headless" title="Postings tagged headless" style="font-size: 10.5px; line-height:10.5px">headless</a> <a href="/tags#homebrew" title="Postings tagged homebrew" style="font-size: 10.5px; line-height:10.5px">homebrew</a> <a href="/tags#javascript" title="Postings tagged javascript" style="font-size: 10.5px; line-height:10.5px">javascript</a> <a href="/tags#jenkins" title="Postings tagged jenkins" style="font-size: 10.5px; line-height:10.5px">jenkins</a> <a href="/tags#osx" title="Postings tagged osx" style="font-size: 10.5px; line-height:10.5px">osx</a> <a href="/tags#qunit" title="Postings tagged qunit" style="font-size: 10.5px; line-height:10.5px">qunit</a> <a href="/tags#ruby" title="Postings tagged ruby" style="font-size: 13.0px; line-height:13.0px">ruby</a> <a href="/tags#selenium" title="Postings tagged selenium" style="font-size: 15.0px; line-height:15.0px">selenium</a> <a href="/tags#testing" title="Postings tagged testing" style="font-size: 20.0px; line-height:20.0px">testing</a> <a href="/tags#ubuntu" title="Postings tagged ubuntu" style="font-size: 10.5px; line-height:10.5px">ubuntu</a> <a href="/tags#vim" title="Postings tagged vim" style="font-size: 11.0px; line-height:11.0px">vim</a> <a href="/tags#webdriver" title="Postings tagged webdriver" style="font-size: 14.5px; line-height:14.5px">webdriver</a> | ||
<a href="/tags#browsermob" title="Postings tagged browsermob" style="font-size: 10.5px; line-height:10.5px">browsermob</a> <a href="/tags#browsermob-proxy" title="Postings tagged browsermob-proxy" style="font-size: 10.5px; line-height:10.5px">browsermob-proxy</a> <a href="/tags#bundler" title="Postings tagged bundler" style="font-size: 10.5px; line-height:10.5px">bundler</a> <a href="/tags#capybara" title="Postings tagged capybara" style="font-size: 11.5px; line-height:11.5px">capybara</a> <a href="/tags#centos" title="Postings tagged centos" style="font-size: 10.5px; line-height:10.5px">centos</a> <a href="/tags#chrome" title="Postings tagged chrome" style="font-size: 10.5px; line-height:10.5px">chrome</a> <a href="/tags#ci" title="Postings tagged ci" style="font-size: 11.0px; line-height:11.0px">ci</a> <a href="/tags#cpaybara" title="Postings tagged cpaybara" style="font-size: 10.5px; line-height:10.5px">cpaybara</a> <a href="/tags#cucumber" title="Postings tagged cucumber" style="font-size: 13.5px; line-height:13.5px">cucumber</a> <a href="/tags#elixir" title="Postings tagged elixir" style="font-size: 10.5px; line-height:10.5px">elixir</a> <a href="/tags#erlang" title="Postings tagged erlang" style="font-size: 10.5px; line-height:10.5px">erlang</a> <a href="/tags#grid" title="Postings tagged grid" style="font-size: 10.5px; line-height:10.5px">grid</a> <a href="/tags#gvim" title="Postings tagged gvim" style="font-size: 10.5px; line-height:10.5px">gvim</a> <a href="/tags#headless" title="Postings tagged headless" style="font-size: 10.5px; line-height:10.5px">headless</a> <a href="/tags#homebrew" title="Postings tagged homebrew" style="font-size: 10.5px; line-height:10.5px">homebrew</a> <a href="/tags#javascript" title="Postings tagged javascript" style="font-size: 10.5px; line-height:10.5px">javascript</a> <a href="/tags#jenkins" title="Postings tagged jenkins" style="font-size: 10.5px; line-height:10.5px">jenkins</a> <a href="/tags#osx" title="Postings tagged osx" style="font-size: 10.5px; line-height:10.5px">osx</a> <a href="/tags#qunit" title="Postings tagged qunit" style="font-size: 10.5px; line-height:10.5px">qunit</a> <a href="/tags#ruby" title="Postings tagged ruby" style="font-size: 13.0px; line-height:13.0px">ruby</a> <a href="/tags#selenium" title="Postings tagged selenium" style="font-size: 15.5px; line-height:15.5px">selenium</a> <a href="/tags#testing" title="Postings tagged testing" style="font-size: 20.0px; line-height:20.0px">testing</a> <a href="/tags#ubuntu" title="Postings tagged ubuntu" style="font-size: 10.5px; line-height:10.5px">ubuntu</a> <a href="/tags#vim" title="Postings tagged vim" style="font-size: 11.0px; line-height:11.0px">vim</a> <a href="/tags#webdriver" title="Postings tagged webdriver" style="font-size: 14.0px; line-height:14.0px">webdriver</a> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,58 @@ | ||
--- | ||
layout: post | ||
title: Rerun failing tests with Cucumber - a solution for nondeterministic UI tests? | ||
author: Matt Robbins | ||
categories: | ||
- testing | ||
- cucumber | ||
- selenium | ||
--- | ||
|
||
The evolution of acceptance test automation, or at least my experience of it has been something like this: | ||
|
||
_2005 - 2008_ : QA runs automated test on their machine pior to raising deployment ticket | ||
|
||
_2009 - 2012_ : CI runs automated tests - downstream, everybody ignores them | ||
|
||
_2012 - present_ : Continuous Delivery...UI tests broken, no deployments...now we have your attention! | ||
|
||
So hopefully you have arrived at stage 3 and have done the following to ensure your UI tests are as resilient as possible: | ||
|
||
* Followed Martin Fowler's [test pyramid](http://martinfowler.com/bliki/TestPyramid.html) and not implemented every conceivable system test via Selenium | ||
* Implemented sane retry logic to find UI elements | ||
* Isolated your UI using stub backends to guard against unexpected data | ||
* Added spoonfuls of helpful debug logging to highlight issues | ||
|
||
But your tests still appear now and again be nondeterministic and people are getting frustrated. | ||
|
||
This can happen, UI testing is hard and no matter how much you defend against it peculiarities of the runtime environment can conspire against you. | ||
|
||
In this instance it might be handy to have 'one more go' when you get some failures and Cucumber's rerun formatter allows this. | ||
|
||
The Example | ||
----------- | ||
|
||
[This](https://github.com/mcrmfc/cucumber-rerun-example) is about the simplest example I could come up with. | ||
|
||
The test will fail 50% of the time, allowing you to see the rerun kicking in on selected runs. | ||
|
||
You can run the test using `bundle exec rake` | ||
|
||
The test should be run for a second time if it fails and should terminate with an appropriate exit code. | ||
|
||
You can see some examples of this working in the project's [Travis](https://travis-ci.org/mcrmfc/cucumber-rerun-example/builds) build. | ||
|
||
Other Thoughts | ||
---------------- | ||
|
||
Another useful thing might be to run any failing tests from previous CI runs first and you could probably adapt this approach to do just that! | ||
|
||
Credit | ||
------ | ||
|
||
I just thought I would illustrate with an example what others have already highlighted: | ||
|
||
<http://blog.mattheworiordan.com/post/42587188567/automatically-retrying-failing-non-deterministic> | ||
|
||
<http://blog.crowdint.com/2011/08/22/auto-retry-failed-cucumber-tests.html> | ||
|
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.