Unit Testing

[grahamboree]

We've talked about adding unit tests to some core components of the project. I thought I'd open a thread to discuss what areas should be prioritized for unit testing and what framework we should use.

Which framework should we use?
Personally, my experiences unit testing C++ haven't been as smooth as in other environments, though it's certainly better than nothing. I've enjoyed using Catch for other projects, as it's lightweight, straightforward, and easy to integrate and work with. Google Test is another reasonable alternative, but it's probably a bit too heavy handed for what we'd need. There are tons of other options as well. Wikipedia has a huge list of C++ unit testing frameworks.

Where do the tests live?
The tests should not interfere at all with the actual library code or the current integration process. This probably means that they should probably be in a location that is completely separate from the codebase so that integration doesn't involve a "remove the tests" step.

The tests should be easy to write and run. This is important in encouraging test creation and maintenance. If the unit test suite is hard to run, they'll be ignored. This assertion is subservient to the current codebase's integration experience. When in doubt, it's better to make the unit testing experience more tedious if in doing so, the experience of using using recast is maintained or improved.

What should we be testing (first)?

A few parts of the codebase are pretty straightforward to test, and would benefit from knowing they're a solid foundation. In particular the majority of geometry and math helper functions are already pure, making testing them trivial. e.g. distancePtSeg, dividePoly, etc...

There are also a lot of small helper functions like insertSortthat are pretty easy to test.

Beyond that, it starts getting into low-level recast functions, which are definitely testable, but might require a bit more domain-knowledge.

[grahamboree]

Another aspect to this we should consider: Once we have some unit tests, we should get them outputting gcov information when the game is built by the CI servers. This will allow us to use things like coveralls to measure code coverage.

Here's an example of a C++ project that has CI with Travis and Appveyor set up, along with unit tests using Catch and code coverage with coveralls. Their unit tests are all in one giant 10k line cpp file, which is less than ideal for sure, but is a good reference point when discussing how to structure our changes.

[hymerman]

Haha, we're going to get along well Graham :) This is something I was considering doing already.

My personal favourite unit test framework for C++ is UnitTest++ (https://github.com/unittest-cpp/unittest-cpp). I'd be perfectly happy with Catch or something else simple too though.

I'm not sure where in the repo they should live - each sub-project (Detour, DetourCrowd etc) is already in its own directory (which is nice), so either a completely parallel 'tests' directory or a 'tests' directory within each sub-project? And yes, easy to run is key - I make unit tests part of my build script. I'm sure we can get premake to run the tests.

Agreed on first things to test.

Also completely agreed on gcov or similar. Should be easy enough to work in. I've also got a long list of fun things I'd like to do with tests (fuzz testing, clang sanitizers etc).

I wonder what we should do with larger-scale functional tests? It would be nice to have automated tests that run through the whole process of building a navmesh, and/or pathfinding on it. Not as part of the unit test suite of course, but given the codebase's functional style it would be simple to use the same framework to write them and just include running them as an optional build target.