Skip to content

Commit

Permalink
new post on cucumber rerun
Browse files Browse the repository at this point in the history
  • Loading branch information
Matthew Robbins committed May 13, 2015
1 parent b0d336a commit 58de26a
Show file tree
Hide file tree
Showing 4 changed files with 153 additions and 186 deletions.
2 changes: 1 addition & 1 deletion _includes/tag_cloud.html
@@ -1 +1 @@
<a href="/tags#browsermob" title="Postings tagged browsermob" style="font-size: 10.5px; line-height:10.5px">browsermob</a> <a href="/tags#browsermob-proxy" title="Postings tagged browsermob-proxy" style="font-size: 10.5px; line-height:10.5px">browsermob-proxy</a> <a href="/tags#bundler" title="Postings tagged bundler" style="font-size: 11.0px; line-height:11.0px">bundler</a> <a href="/tags#capybara" title="Postings tagged capybara" style="font-size: 12.0px; line-height:12.0px">capybara</a> <a href="/tags#centos" title="Postings tagged centos" style="font-size: 10.5px; line-height:10.5px">centos</a> <a href="/tags#chrome" title="Postings tagged chrome" style="font-size: 10.5px; line-height:10.5px">chrome</a> <a href="/tags#ci" title="Postings tagged ci" style="font-size: 11.0px; line-height:11.0px">ci</a> <a href="/tags#cpaybara" title="Postings tagged cpaybara" style="font-size: 10.5px; line-height:10.5px">cpaybara</a> <a href="/tags#cucumber" title="Postings tagged cucumber" style="font-size: 13.0px; line-height:13.0px">cucumber</a> <a href="/tags#elixir" title="Postings tagged elixir" style="font-size: 10.5px; line-height:10.5px">elixir</a> <a href="/tags#erlang" title="Postings tagged erlang" style="font-size: 10.5px; line-height:10.5px">erlang</a> <a href="/tags#grid" title="Postings tagged grid" style="font-size: 10.5px; line-height:10.5px">grid</a> <a href="/tags#gvim" title="Postings tagged gvim" style="font-size: 10.5px; line-height:10.5px">gvim</a> <a href="/tags#headless" title="Postings tagged headless" style="font-size: 10.5px; line-height:10.5px">headless</a> <a href="/tags#homebrew" title="Postings tagged homebrew" style="font-size: 10.5px; line-height:10.5px">homebrew</a> <a href="/tags#javascript" title="Postings tagged javascript" style="font-size: 10.5px; line-height:10.5px">javascript</a> <a href="/tags#jenkins" title="Postings tagged jenkins" style="font-size: 10.5px; line-height:10.5px">jenkins</a> <a href="/tags#osx" title="Postings tagged osx" style="font-size: 10.5px; line-height:10.5px">osx</a> <a href="/tags#qunit" title="Postings tagged qunit" style="font-size: 10.5px; line-height:10.5px">qunit</a> <a href="/tags#ruby" title="Postings tagged ruby" style="font-size: 13.0px; line-height:13.0px">ruby</a> <a href="/tags#selenium" title="Postings tagged selenium" style="font-size: 15.0px; line-height:15.0px">selenium</a> <a href="/tags#testing" title="Postings tagged testing" style="font-size: 20.0px; line-height:20.0px">testing</a> <a href="/tags#ubuntu" title="Postings tagged ubuntu" style="font-size: 10.5px; line-height:10.5px">ubuntu</a> <a href="/tags#vim" title="Postings tagged vim" style="font-size: 11.0px; line-height:11.0px">vim</a> <a href="/tags#webdriver" title="Postings tagged webdriver" style="font-size: 14.5px; line-height:14.5px">webdriver</a>
<a href="/tags#browsermob" title="Postings tagged browsermob" style="font-size: 10.5px; line-height:10.5px">browsermob</a> <a href="/tags#browsermob-proxy" title="Postings tagged browsermob-proxy" style="font-size: 10.5px; line-height:10.5px">browsermob-proxy</a> <a href="/tags#bundler" title="Postings tagged bundler" style="font-size: 10.5px; line-height:10.5px">bundler</a> <a href="/tags#capybara" title="Postings tagged capybara" style="font-size: 11.5px; line-height:11.5px">capybara</a> <a href="/tags#centos" title="Postings tagged centos" style="font-size: 10.5px; line-height:10.5px">centos</a> <a href="/tags#chrome" title="Postings tagged chrome" style="font-size: 10.5px; line-height:10.5px">chrome</a> <a href="/tags#ci" title="Postings tagged ci" style="font-size: 11.0px; line-height:11.0px">ci</a> <a href="/tags#cpaybara" title="Postings tagged cpaybara" style="font-size: 10.5px; line-height:10.5px">cpaybara</a> <a href="/tags#cucumber" title="Postings tagged cucumber" style="font-size: 13.5px; line-height:13.5px">cucumber</a> <a href="/tags#elixir" title="Postings tagged elixir" style="font-size: 10.5px; line-height:10.5px">elixir</a> <a href="/tags#erlang" title="Postings tagged erlang" style="font-size: 10.5px; line-height:10.5px">erlang</a> <a href="/tags#grid" title="Postings tagged grid" style="font-size: 10.5px; line-height:10.5px">grid</a> <a href="/tags#gvim" title="Postings tagged gvim" style="font-size: 10.5px; line-height:10.5px">gvim</a> <a href="/tags#headless" title="Postings tagged headless" style="font-size: 10.5px; line-height:10.5px">headless</a> <a href="/tags#homebrew" title="Postings tagged homebrew" style="font-size: 10.5px; line-height:10.5px">homebrew</a> <a href="/tags#javascript" title="Postings tagged javascript" style="font-size: 10.5px; line-height:10.5px">javascript</a> <a href="/tags#jenkins" title="Postings tagged jenkins" style="font-size: 10.5px; line-height:10.5px">jenkins</a> <a href="/tags#osx" title="Postings tagged osx" style="font-size: 10.5px; line-height:10.5px">osx</a> <a href="/tags#qunit" title="Postings tagged qunit" style="font-size: 10.5px; line-height:10.5px">qunit</a> <a href="/tags#ruby" title="Postings tagged ruby" style="font-size: 13.0px; line-height:13.0px">ruby</a> <a href="/tags#selenium" title="Postings tagged selenium" style="font-size: 15.5px; line-height:15.5px">selenium</a> <a href="/tags#testing" title="Postings tagged testing" style="font-size: 20.0px; line-height:20.0px">testing</a> <a href="/tags#ubuntu" title="Postings tagged ubuntu" style="font-size: 10.5px; line-height:10.5px">ubuntu</a> <a href="/tags#vim" title="Postings tagged vim" style="font-size: 11.0px; line-height:11.0px">vim</a> <a href="/tags#webdriver" title="Postings tagged webdriver" style="font-size: 14.0px; line-height:14.0px">webdriver</a>
58 changes: 58 additions & 0 deletions _posts/2015-05-13-cucumber-rerun-formatter.markdown
@@ -0,0 +1,58 @@
---
layout: post
title: Rerun failing tests with Cucumber - a solution for nondeterministic UI tests?
author: Matt Robbins
categories:
- testing
- cucumber
- selenium
---

The evolution of acceptance test automation, or at least my experience of it has been something like this:

_2005 - 2008_ : QA runs automated test on their machine pior to raising deployment ticket

_2009 - 2012_ : CI runs automated tests - downstream, everybody ignores them

_2012 - present_ : Continuous Delivery...UI tests broken, no deployments...now we have your attention!

So hopefully you have arrived at stage 3 and have done the following to ensure your UI tests are as resilient as possible:

* Followed Martin Fowler's [test pyramid](http://martinfowler.com/bliki/TestPyramid.html) and not implemented every conceivable system test via Selenium
* Implemented sane retry logic to find UI elements
* Isolated your UI using stub backends to guard against unexpected data
* Added spoonfuls of helpful debug logging to highlight issues

But your tests still appear now and again be nondeterministic and people are getting frustrated.

This can happen, UI testing is hard and no matter how much you defend against it peculiarities of the runtime environment can conspire against you.

In this instance it might be handy to have 'one more go' when you get some failures and Cucumber's rerun formatter allows this.

The Example
-----------

[This](https://github.com/mcrmfc/cucumber-rerun-example) is about the simplest example I could come up with.

The test will fail 50% of the time, allowing you to see the rerun kicking in on selected runs.

You can run the test using `bundle exec rake`

The test should be run for a second time if it fails and should terminate with an appropriate exit code.

You can see some examples of this working in the project's [Travis](https://travis-ci.org/mcrmfc/cucumber-rerun-example/builds) build.

Other Thoughts
----------------

Another useful thing might be to run any failing tests from previous CI runs first and you could probably adapt this approach to do just that!

Credit
------

I just thought I would illustrate with an example what others have already highlighted:

<http://blog.mattheworiordan.com/post/42587188567/automatically-retrying-failing-non-deterministic>

<http://blog.crowdint.com/2011/08/22/auto-retry-failed-cucumber-tests.html>

99 changes: 0 additions & 99 deletions sitemap.xml

This file was deleted.

0 comments on commit 58de26a

Please sign in to comment.