Acceptance or end-to-end tests and QA

Open to constructive criticism and improvement!

So far, only the google test framework and the code coverage provide support for unit tests. Although we have some other tests, these are not continuously running.

Testing of the "end-to-end" functionality is an important aspect. With this milestone done, all of the unit tests, acceptance (or end-to-end) tests, their code coverage and simple code analysis for QA (like trailing whitespaces function names and some simple metrics like function lengths) will be provided.

Based on our experience with OGS5 benchmarks, Lars and I propose the following basis for the testing environment:

• User provides simple shells script to prepare, run and postprocess a benchmark in these three parts: pre(), run(), post().
• These shell scripts are called from a testing application, which collects the output. The testing application can be called in CI environment or by command line.
• The output can be further processed by a web server to publish the results in an illustrative chart (with green/red markers etc.).

Further applications include:

• Code coverage of the acceptance tests (together or separately with unit tests)
• Running Doxygen to discover undocumented interfaces.
• Analyse code for simple formatting issues or metrics, like function lengths.

Later, should the need arise, we could specialise any of the pre(), run() or post() functions for running FEM benchmarks, for example.

Another development direction would be to analyse the output of the testing application in a more detailed way, e.g. the numerical deviation of an FEM benchmark from the analytical solution.

ChangeLog:
31. Oct. 2012 - Meeting Dima and Lars
12. Nov. 2012 - Spell check, Dima
03. Dec. 2012 - Change due dates, because still working on basic libs. Dima