Running and creating tests

Camdram has a growing but still somewhat patchy set of automated tests. These are used to verify that the features covered work as expected and have not unexpectedly regressed.

These run automatically on Travis every time new commits are pushed to Github, and you will receive an email notification if the tests fail. It is often a good idea to run the tests locally too before committing any significant changes.

All tests make use of the PHPUnit library. Functional tests (which simulate full web requests and validate the output) addtionally make use of functionality built in the Symfony framework, detailed here.

All tests live in the tests/ folder, using a folder structure that mirrors the contents of src/. The unit test for TextService is one of the simpler examples to get started with.

The tests use an SQLite database, and the DoctrineTestBundle is used to start a new transaction at the start of each test and roll it back at the end so that each test starts with a clean slate. A custom bootstrap file (tests/boostrap.php) creates the initial schema (if it doesn't already exist) but no data is populated - each test is responsible for creating the fixtures it requires.

Running tests

The complete set of checks that run on Travis (including YAML and Twig file validation) can be run using:

./runtests

Alternatively, just the PHPUnit tests can be run using:

php bin/simple-phpunit

If you're only working on a specific test case or set of test cases, a filename or subfolder can be passed as an argument, e.g.:

php bin/simple-phpunit tests/CamdramBundle/Controller

Types of test

Broadly speaking, tests fall into two types: unit tests and functional tests.

Unit tests validate a small part of the source code (normally a single class). Sometimes, these 'mock out' the classes it depends on in order to isolate the functionality being tested from bugs that might exist in other code, the database or the network. This means they generally run very quickly and are useful for locally testing more complex pieces of functionality in isolation.

Unit tests can also be a form of documentation of the underlying code (e.g. if you wondered what exactly the 'detectLinks' method does in TextService you could look at e.g. the 'testDetectLinks_Urls' method of its test).

Functional tests are useful as they allow whole features and UI flows to be tested end-to-end, but they are a little slower than unit tests. Symfony comes with utilities to automatically drive the website as a user would and test for certain outcomes (e.g. there's a test that fills in the login form then ensures that you are logged in). They can be more brittle because they depend on the entire stack of dependencies (e.g. if there's a problem with the database then all the tests will fail even though the problem lies elsewhere).

Travis CI

./runtests is run automatically by Travis CI each time someone pushes new code to Github. The results and the status of any running tests can be viewed at https://travis-ci.org/camdram/camdram.

./check_dev_setup also runs on Travis, which runs against a MySQL database. This validates certain common commands used by developers and ensures that the DB migrations are valid and in sync with the entity definitions.

If one of your commits causes the tests to fail, Travis CI will send you an e-mail notifying you, which will contain a link to the output of the failed tests. If this happens, it is likely that your work has caused a regression somewhere - it should be possible to work out the problem by browsing the output of the tests. If it's not obvious then contact another Camdram dev or raise another issue on Github.

Travis is configured using .travis.yml configuration file in the root of the repository.