diff --git a/_includes/tag_cloud.html b/_includes/tag_cloud.html index 41cb12d..b2803f3 100644 --- a/_includes/tag_cloud.html +++ b/_includes/tag_cloud.html @@ -1 +1 @@ -browsermob browsermob-proxy bundler capybara centos chrome ci cpaybara cucumber elixir erlang grid gvim headless homebrew javascript jenkins osx qunit ruby selenium testing ubuntu vim webdriver +browsermob browsermob-proxy bundler capybara centos chrome ci cpaybara cucumber elixir erlang grid gvim headless homebrew javascript jenkins osx qunit ruby selenium testing ubuntu vim webdriver diff --git a/_posts/2015-05-13-cucumber-rerun-formatter.markdown b/_posts/2015-05-13-cucumber-rerun-formatter.markdown new file mode 100644 index 0000000..be2dfbe --- /dev/null +++ b/_posts/2015-05-13-cucumber-rerun-formatter.markdown @@ -0,0 +1,58 @@ +--- +layout: post +title: Rerun failing tests with Cucumber - a solution for nondeterministic UI tests? +author: Matt Robbins +categories: +- testing +- cucumber +- selenium +--- + +The evolution of acceptance test automation, or at least my experience of it has been something like this: + +_2005 - 2008_ : QA runs automated test on their machine pior to raising deployment ticket + +_2009 - 2012_ : CI runs automated tests - downstream, everybody ignores them + +_2012 - present_ : Continuous Delivery...UI tests broken, no deployments...now we have your attention! + +So hopefully you have arrived at stage 3 and have done the following to ensure your UI tests are as resilient as possible: + +* Followed Martin Fowler's [test pyramid](http://martinfowler.com/bliki/TestPyramid.html) and not implemented every conceivable system test via Selenium +* Implemented sane retry logic to find UI elements +* Isolated your UI using stub backends to guard against unexpected data +* Added spoonfuls of helpful debug logging to highlight issues + +But your tests still appear now and again be nondeterministic and people are getting frustrated. + +This can happen, UI testing is hard and no matter how much you defend against it peculiarities of the runtime environment can conspire against you. + +In this instance it might be handy to have 'one more go' when you get some failures and Cucumber's rerun formatter allows this. + +The Example +----------- + +[This](https://github.com/mcrmfc/cucumber-rerun-example) is about the simplest example I could come up with. + +The test will fail 50% of the time, allowing you to see the rerun kicking in on selected runs. + +You can run the test using `bundle exec rake` + +The test should be run for a second time if it fails and should terminate with an appropriate exit code. + +You can see some examples of this working in the project's [Travis](https://travis-ci.org/mcrmfc/cucumber-rerun-example/builds) build. + +Other Thoughts +---------------- + +Another useful thing might be to run any failing tests from previous CI runs first and you could probably adapt this approach to do just that! + +Credit +------ + +I just thought I would illustrate with an example what others have already highlighted: + + + + + diff --git a/sitemap.xml b/sitemap.xml deleted file mode 100644 index e6b4397..0000000 --- a/sitemap.xml +++ /dev/null @@ -1,99 +0,0 @@ - - - - http://opensourcetester.co.uk/about.html - 2013-07-22 - - - http://opensourcetester.co.uk/atom.xml - 2013-07-22 - - - http://opensourcetester.co.uk/ - 2013-09-23 - - - http://opensourcetester.co.uk/tags.html - 2015-01-18 - - - http://opensourcetester.co.uk/2015/01/16/from-zero-to-tested-with-centos7-and-chrome - 2015-01-16 - - - http://opensourcetester.co.uk/2013/08/15/selenium-141-browsermob-proxy-alternatives - 2013-08-15 - - - http://opensourcetester.co.uk/2013/08/06/erlangr16-osx - 2013-08-06 - - - http://opensourcetester.co.uk/2013/06/28/jenkins-windows-ssh - 2013-06-28 - - - http://opensourcetester.co.uk/2012/11/29/zeros-padding-3des-ruby-openssl - 2012-11-29 - - - http://opensourcetester.co.uk/2012/11/26/two-level-gherkin-indent-vim - 2012-11-26 - - - http://opensourcetester.co.uk/2012/08/06/cucumber-page-object-global - 2012-08-06 - - - http://opensourcetester.co.uk/2012/08/03/pass-parameters-hudson - 2012-08-03 - - - http://opensourcetester.co.uk/2012/03/23/webdriver-modaldialog - 2012-03-23 - - - http://opensourcetester.co.uk/2012/01/04/browsermob-through-proxy - 2012-01-04 - - - http://opensourcetester.co.uk/2011/11/24/cucumber-requires-magic - 2011-11-24 - - - http://opensourcetester.co.uk/2011/11/08/selenium-webdriver-useragent - 2011-11-08 - - - http://opensourcetester.co.uk/2011/10/01/maximise-gvim - 2011-10-01 - - - http://opensourcetester.co.uk/2011/09/13/rubygems-over-ssl-clientcert - 2011-09-13 - - - http://opensourcetester.co.uk/2011/07/11/selenium-webdriver-focus - 2011-07-11 - - - http://opensourcetester.co.uk/2011/07/06/selenium-grid-2 - 2011-07-06 - - - http://opensourcetester.co.uk/2011/06/30/qunit-selenium-sauce - 2011-06-30 - - - http://opensourcetester.co.uk/2011/06/24/capybara-android - 2011-06-24 - - - http://opensourcetester.co.uk/2011/06/23/capybara-webkit-centos - 2011-06-23 - - - http://opensourcetester.co.uk/2011/06/13/first - 2011-06-13 - - \ No newline at end of file diff --git a/tags.html b/tags.html index ceff3bb..11b4170 100644 --- a/tags.html +++ b/tags.html @@ -25,22 +25,18 @@

browsermob

13 Sep 2011
How to serve rubygems over ssl using client certification and Bundler -
  • -
    13 Sep 2011
    - How to serve rubygems over ssl using client certification and Bundler -

    • capybara

    centos

    ci

    cpaybara

    cucumber

    elixir

    ruby

    selenium

    testing

    ubuntu

    vim

    webdriver