The proof of concept at https://github.com/silverstripe-labs/spec-by-example already implements some basic steps. It will be very important to implement the most common basic steps and make them as versatile as possible. The developers shouldn't reinvent the wheel while writing acceptance tests for their own modules.
This also highlights the need of documentation - a part of this task will be to set up a doc for developers to make it clear, what methods are available to use.
I managed to set up the environment on my machine and I can see some field for improvements: first of all, SilverStripe integration which I already describe in the last paragraph. Secondly, the test runner should be "plug and play" - we need something that requires one command to set up the server, preferably set up a clear SilverStripe instance for every test suite, run the tests, shut down the server.
My point is, that while it is a good idea to test on different browsers, running the tests four times on every commit may be very resource-consuming. Selenium tests take much longer than simple unit tests and we wouldn't want to slaughter the server with endless test requests. It is up for a discussion, but maybe it would be smart to test only on Chrome Driver (it has the most developed features) and test on all browsers once a day or so.
It is very important to involve the community into acceptance tests, especially because almost anyone can write the user stories. This way, the programmers only need to cover the stories in format "When I click (this), then (that) happens" - it should be very helpful when the testing project goes bigger.
Also, currently the docs are meant for Mac users, which requires additional effort from Linux users. The new, improved docs will be cross-platform and split in two parts: one strictly for developers, and one related to user stories
This is a very scalable part, which we can extend or minimize depending on the time which it will take to implement the rest. We can just allow adding acceptance tests to modules and stop at this point at the end of GSoC.
However, if there is enough time, we could allow some very interesting improvements: for example, themes could come with optionally bundled acceptance tests. This would allow to make sure that their basic functionalities aren't broken by other widgets or modules. Same applies for widgets. It's just a thought, but I think that it sounds like a good idea for future.
SilverStripe currently uses buildbot and TeamCity for CI. It would be ideal to run all tests from the same place. One of the possible last parts of the project will be integrating the CI environment with acceptance tests to have ALL the tests run at once.
After quickly grasping through SilverStripe unit tests and the spec-by-example project, I noticed one thing: the current implementation doesn't take SilverStripe integration into account. By that, I mean that the tests are ran outside of SilverStripe and all actions have to be performed in a "behavioural" way.
This leads to some unwanted consequences: for example, if we wanted to test a pagination feature, we would have to schedule adding multiple users "by hand", and the Web Driver would take a lot of time to perform these actions. Imagine adding 50 or 100 users by clicking through the whole interface! Even your computer would sweat from this.
Instead, a good idea would be to integrate the acceptance tests with SilverStripe and make use of a Database Factory, such as Phactory: http://phactory.org/. What does it do? Basically, it allows to generate some Model data in a very easy way from the backend, without performing actual behavioral actions on the frontend and without executing complex functions on the backend.
Models are often linked with each others and factories can take care of that as well, they can include each other to generate a cascade of database entries in a blitz.
How does it improve acceptance tests? Basically, if we want to test pagination or some other feature requiring lots of records, we just run the factories inside acceptance tests first, and then perform the actual behavioral actions after the database is filled. It saves a lot of time every time the tests are ran.
This point will be probably discussed later, because I'm not sure about how easily SilverStripe could be integrated with the tests - this would require som research and experiments but I think that it would be highly beneficial.
NaturalWebdriverwhich are not available through
WebDriverSteps) to behat
FeatureContext(and split into subcontexts)
NaturalWebDriver->ajaxClickHandler_before()jQuery monkeypatch to behat, confirm it works