The examples here cover three different problems:
- testing REST APIs
- testing web pages
- simplifying writing of tests.
Tools for writing REST APIs make it easy to do HTTP calls and provide ways of writing assertions about JSON or XML documents.
The rest-assured tool looks pretty good tool to do this kind of thing. Rest-Driver developed by Nokia Music or RESTFuse do similar things. There are several blog posts about how to test REST APIs with Spock or Cumcumber-JVM. They use tools such as JsonAssert although only solves part of the problem.
For testing web pages, people generally use selenium / Web driver. However there are three problems with this tool:
- The Selenium browser plugins can be unreliable or difficult to configure on a continuous integration machine. Therefore people tend to prefer HtmlUnit, which is pure Java so runs fast, but isn't based on a real browser so has some strange behaviors, or phantomjs which is based on Webkit but is a bit faster / more reliable than an embedded browser. The PhantomJS Maven plugin makes deploying PhantomJS easier. PhantomJS is compatible with Selenium/Webdriver so you can still write code in the same way.
- The Selenium / Webdriver API is low level so writing tests takes time. So there are several different tools that try to improve on this e.g. fluentlenium or selenide. geb which uses Groovy could also be put in this category. None of these tools seem to be perfect although these three are probably the most mature, although Geb is probably the least mature of the three. They are still not totally satisfactory: for example Fluentlenium doesn't seem to work with PhantomJS - David Gageot has published some gists 1 2 and written a project called Simplelenium to address this although it is still at the prototype stage. Selenide does [seem to support PhantomJS}(http://selenide.org/javadoc/2.5/com/codeborne/selenide/Configuration.html#browser).
- Sometimes the tests are written by non-coders who tend to use the seleniumide tool. This is a plugin for Firefox that records someone driving a web browser, producing an Expect-like script in XML. This allows you to write tests quickly, but sometimes those tests are brittle and it can be hard to do large scale automation. The https://code.google.com/p/selunit/ project aims to let you run these scripts from JUnit / Maven, so you can run them with CI. SauceLabs also have Selenium Builder which is a better version of SeleniumIDE.
Test code can get ugly quick, it has a maintenance cost and it can impact the speed of code development because badly designed tests mean changing the main code requires lots of test code changes.
Although behavior driven design tools are touted as a different "behavior driven" approach to creating software, they can also help with simplifying the test code and making it more maintainable.
There is a nice post on Java ranch that makes a distinction between higher level BDD tools such as JBehave, cucumber and Cucumber-JVM, EasyB and Thucydides and TDD-style tools such as Spock or Specs2.
Testatoo is a high level BDD tool written in Java that is designed explicitly to write tests for Selenium / Webdriver.
There are blog posts with examples of using both Cucumber and Spock to test REST APIs and web pages:
- Using Cucumber-JVM to test REST APIs
- Cucumber and rest-assured
- Using Cucumber-JVM with Selenium
- Using Spock to test REST apis
- Using Spock and Selenium
- Using Spock with Geb
If you are prepared to go with non-JVM based solutions then there are possibilites such as:
- robot framework Testing framework normally used with Python although you can use it with other languages. Companies like Nokia and Verisign use this framework.
- Jasmine Popular BDD framework for Javascript.
- frisby.js A node.js based framework for testing REST APIs that uses jasmine.
- cucumber
- Ruby rest-assured Confusingly the BBC developed a different Ruby library also called rest-assured.
Here are two patterns that might be relevant to the testing REST APIs and web page tasks:
One pattern I used a lot in web crawling and data extraction framework I wrote I used "golden files" for test cases.
Here when testing an operation I have files that record the input and output. For each test, the test runner runs the test, but if a canonical result (the "golden file") does not exist it, it saves the output as a "candidate" golden file. Then the tester can inspect the output, check it is correct. Once the output is accepted, it is committed to source control. This is primarily a way of checking further changes don't cause regressions.
Obviously this isn't as flexible as some other approaches, but it is possible to write a lot of test cases quickly using the pattern. Also I like being able to "eyeball" the expected result - sometimes that's easier then looking at some test code and having to remember what it is testing for.
There are some small niggles with this approach: if you are using time / timestamping in the result you need to filter that out, and also you need a canonical serialization order for the JSON / XML.
I have a custom JUnit runner to support this approach I can share.
When testing the deployed app, If the deployed app is a live service, then the data is changing. A common idea in the ROR world is a fixture which means sample data. So it might be helpful to deploy a version of the latest code against a fixture. This can might make testing easier and more repeatable. It also lets you tear down the fixture and repopulate the data at the end of the run.
There are a few blog posts that compare different frameworks:
I have put together some simple "hello-world" examples for
- Testing the Youtube webservice using rest-assured.
- Testing Google.com using PhantomJS and the basic Selenium / Webdriver API.
- Testing Bing.com using Fluentlenium, a wrapper around Selenium / Webdriver.
- Testing the Youtube API using Cucumber-JVM / rest-assured.
- Testing the Bing.com using Cucumber-JVM, PhantomJS, Selenium / Webdriver API.
So clearly it is possible to combine these different tools together.
I have also collected links to different articles about these tools and other tools mentioned above in the README.md files in the subdirectories.