End-to-end test framework

[andorp]

Currently there are simple unit tests as part of the testing process. I would like to extend this with an end-to-end approach similar I implemented in the idris-grin test library.

The current situation we have:
We run unit tests using Hspec and report the coverage of the result. After the unit tests we run the grin compiler on the sum_simple example. The build fails if there are test failures or some exception during the compilation of the sum_simple.grin. The output of the sum_simple compilation is never checked. As the project grows an gains popularity and user base we need a more thorough testing.

The use-case for this extension is already discussed:

• It would be good to maintain a set of example grin files in the repository and use them during the development and after during part of the regression testing
• Test cases from x-grin back-ends would be easily added to the future regression suite

I propose the following: The end-to-end test suite should

• run after the unit test phase
• have a test data directory where all the test grin files should live
• be able to read the grin files from textual or binary format
• grin files which have main modules should be interpreted (see details below)
• grin files which does not have main should have an option yaml file which sets the pipeline options and an expected output file
• add to the coverage report

Interpreted tests: The end-to-end suite should

• run the interpreter on the test grin file without any optimizations, save the output as expected result
• run the interpreter on the test grin file with the optimization pipeline, and compare the output with the expected output. If there is a difference it should bisect the intermediate optimization steps to pinpoint the where the output started to diverge from the expected.
• generate an executable from the optimized version and check the output against the expected.

Thoughts on the binary format: Binary input format is not preferred. Currently it is a short cut to overcome some issues in the parser. Backward compatibility for the end-to-end tests are not required. Whenever we will need to use binary format in the end-to-end test suite it indicates some regression in the parser. After fixing the regression in the parser the binary stored test data should be converted to textual grin. This can be done using the grin compiler.

[andorp]

Please review and comment.