Set up testing and benchmarking framework

[rcurtin]

Reported by rcurtin on 27 Jul 40301847 09:46 UTC
This is a master ticket for a very big, time-consuming task.

The end goal of this is to have a set of tests that are run by the automatic build server once the current svn build is run. These tests will be unit tests as well as tests of the MLPACK methods. Timing information should be collected so that speed improvements (or the lack of them) can be seen.

However it should also be trivial for a user to run these tests. That is, if they type make test, the tests should all be run. This will require including the necessary datasets in the build, and making sure that the information for each test to be run is there.

As it currently stands, a lot of the MLPACK methods build a test executable which is supposed to be run with some set of parameters, but a lot of them don't include the right datasets in the right places or document how to use them. Anyways the intricacies of these tests need to be documented, and CMake targets to actually run the tests should be included (this should not be hard although it may be difficult to make that cross-platform and it may be that that is not feasible in the end).

Boost has a unit testing framework which warrants some investigation here (or so I hear).

Migrated-From: http://trac.research.cc.gatech.edu/fastlab/ticket/19

[rcurtin]

Commented by speet3 on 18 Apr 41592779 02:53 UTC
(In #94) It appears I do not have the correct permission to modify the description of the ticket, so I'll just add it as a comment for now:

The following requirements are excerpted from ticket #18 for as a reminder of intended result of this ticket:

• Build each revision of MLPACK shortly after it is checked in
• Upon encountering a build error that was not present before, alert the developer who checked in the erroneous code with a nastygram
• Be able to run unit tests on the code
• Be able to run timing tests on the code (Remove fl-build system (all the build.py files) #19)
• Be able to get profiling information for the code; CMake can compile MLPACK easily with profiling information (though I've not yet checked the code to make that change in) and then gprof can get results (Minimize preprocessor macro usage #20)
• Compare timing information and profiling information across revisions of the code
• Produce working executables for Windows, Linux, and OS X
• Provide automatically built installation packages

[rcurtin]

Since we don't have master tickets, this ticket is no longer useful.