GSOC 2012 Project: Improve Behaviour Testing Framework and Test Suite
Clone this wiki locally
Basic steps implementation
The proof of concept at https://github.com/silverstripe-labs/spec-by-example already implements some basic steps. It will be very important to implement the most common basic steps and make them as versatile as possible. The developers shouldn't reinvent the wheel while writing acceptance tests for their own modules.
This also highlights the need of documentation - a part of this task will be to set up a doc for developers to make it clear, what methods are available to use.
Improving the test runner
I managed to set up the environment on my machine and I can see some field for improvements: first of all, SilverStripe integration which I already describe in the last paragraph. Secondly, the test runner should be "plug and play" - we need something that requires one command to set up the server, preferably set up a clear SilverStripe instance for every test suite, run the tests, shut down the server.
My point is, that while it is a good idea to test on different browsers, running the tests four times on every commit may be very resource-consuming. Selenium tests take much longer than simple unit tests and we wouldn't want to slaughter the server with endless test requests. It is up for a discussion, but maybe it would be smart to test only on Chrome Driver (it has the most developed features) and test on all browsers once a day or so.
It is very important to involve the community into acceptance tests, especially because almost anyone can write the user stories. This way, the programmers only need to cover the stories in format "When I click (this), then (that) happens" - it should be very helpful when the testing project goes bigger.
Also, currently the docs are meant for Mac users, which requires additional effort from Linux users. The new, improved docs will be cross-platform and split in two parts: one strictly for developers, and one related to user stories
Support for external modules, themes and widgets testing
This is a very scalable part, which we can extend or minimize depending on the time which it will take to implement the rest. We can just allow adding acceptance tests to modules and stop at this point at the end of GSoC.
However, if there is enough time, we could allow some very interesting improvements: for example, themes could come with optionally bundled acceptance tests. This would allow to make sure that their basic functionalities aren't broken by other widgets or modules. Same applies for widgets. It's just a thought, but I think that it sounds like a good idea for future.
SilverStripe currently uses buildbot and TeamCity for CI. It would be ideal to run all tests from the same place. One of the possible last parts of the project will be integrating the CI environment with acceptance tests to have ALL the tests run at once.
After quickly grasping through SilverStripe unit tests and the spec-by-example project, I noticed one thing: the current implementation doesn't take SilverStripe integration into account. By that, I mean that the tests are ran outside of SilverStripe and all actions have to be performed in a "behavioural" way.
This leads to some unwanted consequences: for example, if we wanted to test a pagination feature, we would have to schedule adding multiple users "by hand", and the Web Driver would take a lot of time to perform these actions. Imagine adding 50 or 100 users by clicking through the whole interface! Even your computer would sweat from this.
Instead, a good idea would be to integrate the acceptance tests with SilverStripe and make use of a Database Factory, such as Phactory: http://phactory.org/. What does it do? Basically, it allows to generate some Model data in a very easy way from the backend, without performing actual behavioral actions on the frontend and without executing complex functions on the backend.
Models are often linked with each others and factories can take care of that as well, they can include each other to generate a cascade of database entries in a blitz.
How does it improve acceptance tests? Basically, if we want to test pagination or some other feature requiring lots of records, we just run the factories inside acceptance tests first, and then perform the actual behavioral actions after the database is filled. It saves a lot of time every time the tests are ran.
This point will be probably discussed later, because I'm not sure about how easily SilverStripe could be integrated with the tests - this would require som research and experiments but I think that it would be highly beneficial.
- Until 20. May - Iterating over the details of the projects, discussing possible ways to integrate SilverStripe with the acceptance tests, discussing CI integration and maybe introducing some improvements to SilverStripe unit tests.
- 20. May - 5. June - Integrating the acceptance tests with SilverStripe, writing first basic steps and using them in first acceptance tests.
- 5. June - 20. June - introducing basic improvements to the test runner and writing more extensive acceptance tests.
- 20. June - 5. July - by the GSoC milestone the acceptance tests will be integrated with SilverStripe and the basic functionalities of SilverStripe will be tested on a high level. The second part of the project will consist of writing detailed acceptance tests, making sure that external unit tests from modules are supported, polishing the docs and working on CI integration.
- 5. July - 20. July - writing detailed acceptance tests and iterating over the test runner. I think that we will be constantly working on the test runner through the whole project to make testing as easy as possible.
- 20. July - 5. August - Preparing docs for developers (basic steps and external module tests), researching CI integration, writing even more acceptance tests if there are some left!
- 5. August until the end of GSoC - working as hard as possible on CI integration, making the best out of acceptance tests bundled with themes.
Migration from Cuke4PHP to Behat
- Adapt parts of
NaturalWebdriverwhich are not available through
- Convert existing step definitions (
WebDriverSteps) to behat
FeatureContext(and split into subcontexts)
NaturalWebDriver->ajaxClickHandler_before()jQuery monkeypatch to behat, confirm it works
- Document setup, create installer shell script which works on standard SS3 checkout
- Student: Michał Ochman (ochman.d.michal at gmail dot com)
- Mentor: Ingo Schommer (ingo at silverstripe dot com)
Status and Source
- "silverstripe-behat-extension" module
- Behat tests in "framework" module
- Behat tests in "cms" module
- Student github account
- initial discussion on google groups
- current discussion on google groups