GSOC 2012 Project: Improve Behaviour Testing Framework and Test Suite

chillu edited this page Nov 16, 2012 · 9 revisions


Basic steps implementation

The proof of concept at already implements some basic steps. It will be very important to implement the most common basic steps and make them as versatile as possible. The developers shouldn't reinvent the wheel while writing acceptance tests for their own modules.

This also highlights the need of documentation - a part of this task will be to set up a doc for developers to make it clear, what methods are available to use.

Improving the test runner

I managed to set up the environment on my machine and I can see some field for improvements: first of all, SilverStripe integration which I already describe in the last paragraph. Secondly, the test runner should be "plug and play" - we need something that requires one command to set up the server, preferably set up a clear SilverStripe instance for every test suite, run the tests, shut down the server.

WebDriver choice

WebDriver provides four APIs: Firefox Driver, Chrome Driver, IExplorer driver and HTMLunit Driver. In general, their goal is to cross-test the same code on different browser. However, from my experience Chrome Driver is much more mature than the rest of them - for example, Firefox Driver has big problems in dealing with DOM modifications - during one of my projects I stumbled upon a situation, where the autosuggestion feature couldn't be tested via Firefox Driver because it didn't detect the new elements, basically it did not refresh the DOM so the JavaScript actions weren't noticed.

My point is, that while it is a good idea to test on different browsers, running the tests four times on every commit may be very resource-consuming. Selenium tests take much longer than simple unit tests and we wouldn't want to slaughter the server with endless test requests. It is up for a discussion, but maybe it would be smart to test only on Chrome Driver (it has the most developed features) and test on all browsers once a day or so.

Docs improvements

It is very important to involve the community into acceptance tests, especially because almost anyone can write the user stories. This way, the programmers only need to cover the stories in format "When I click (this), then (that) happens" - it should be very helpful when the testing project goes bigger.

Also, currently the docs are meant for Mac users, which requires additional effort from Linux users. The new, improved docs will be cross-platform and split in two parts: one strictly for developers, and one related to user stories

Support for external modules, themes and widgets testing

This is a very scalable part, which we can extend or minimize depending on the time which it will take to implement the rest. We can just allow adding acceptance tests to modules and stop at this point at the end of GSoC.

However, if there is enough time, we could allow some very interesting improvements: for example, themes could come with optionally bundled acceptance tests. This would allow to make sure that their basic functionalities aren't broken by other widgets or modules. Same applies for widgets. It's just a thought, but I think that it sounds like a good idea for future.

CI integration

SilverStripe currently uses buildbot and TeamCity for CI. It would be ideal to run all tests from the same place. One of the possible last parts of the project will be integrating the CI environment with acceptance tests to have ALL the tests run at once.

SilverStripe integration

After quickly grasping through SilverStripe unit tests and the spec-by-example project, I noticed one thing: the current implementation doesn't take SilverStripe integration into account. By that, I mean that the tests are ran outside of SilverStripe and all actions have to be performed in a "behavioural" way.

This leads to some unwanted consequences: for example, if we wanted to test a pagination feature, we would have to schedule adding multiple users "by hand", and the Web Driver would take a lot of time to perform these actions. Imagine adding 50 or 100 users by clicking through the whole interface! Even your computer would sweat from this.

Instead, a good idea would be to integrate the acceptance tests with SilverStripe and make use of a Database Factory, such as Phactory: What does it do? Basically, it allows to generate some Model data in a very easy way from the backend, without performing actual behavioral actions on the frontend and without executing complex functions on the backend.

Models are often linked with each others and factories can take care of that as well, they can include each other to generate a cascade of database entries in a blitz.

How does it improve acceptance tests? Basically, if we want to test pagination or some other feature requiring lots of records, we just run the factories inside acceptance tests first, and then perform the actual behavioral actions after the database is filled. It saves a lot of time every time the tests are ran.

This point will be probably discussed later, because I'm not sure about how easily SilverStripe could be integrated with the tests - this would require som research and experiments but I think that it would be highly beneficial.


  • Until 20. May - Iterating over the details of the projects, discussing possible ways to integrate SilverStripe with the acceptance tests, discussing CI integration and maybe introducing some improvements to SilverStripe unit tests.
  • 20. May - 5. June - Integrating the acceptance tests with SilverStripe, writing first basic steps and using them in first acceptance tests.
  • 5. June - 20. June - introducing basic improvements to the test runner and writing more extensive acceptance tests.
  • 20. June - 5. July - by the GSoC milestone the acceptance tests will be integrated with SilverStripe and the basic functionalities of SilverStripe will be tested on a high level. The second part of the project will consist of writing detailed acceptance tests, making sure that external unit tests from modules are supported, polishing the docs and working on CI integration.
  • 5. July - 20. July - writing detailed acceptance tests and iterating over the test runner. I think that we will be constantly working on the test runner through the whole project to make testing as easy as possible.
  • 20. July - 5. August - Preparing docs for developers (basic steps and external module tests), researching CI integration, writing even more acceptance tests if there are some left!
  • 5. August until the end of GSoC - working as hard as possible on CI integration, making the best out of acceptance tests bundled with themes.

Migration from Cuke4PHP to Behat

  • Adapt parts of NaturalWebdriver which are not available through Mink (e.g. isCurrentURLSimilarTo())
  • Convert existing step definitions (WebDriverSteps) to behat FeatureContext (and split into subcontexts)
  • Add NaturalWebDriver->ajaxClickHandler_before() jQuery monkeypatch to behat, confirm it works
  • Document setup, create installer shell script which works on standard SS3 checkout


  • Student: Michał Ochman (ochman.d.michal at gmail dot com)
  • Mentor: Ingo Schommer (ingo at silverstripe dot com)

Status and Source