Better integration with tasty

[Bodigrim]

Thanks for creating this library, looks very interesting!

Could the integration with tasty be improved? At the moment tasty-coverage ignores almost everything and resorts to its own launcher, which is not quite sufficient in general. I think one can write an ingredient transformer, which takes a TestTree, wraps every SingleTest into withResource initializing and releasing .tix data and passes the result to another ingredient (e. g., standard console reporter or whatever).

[BinderDavid]

I am definitely looking for better integration with tasty in the long run. One of the problems I foresee is that the --report-coverage function needs to disable parallel test execution. Otherwise the generated .tix files will probably be useless, since tests which are executed in parallel will write simultaneously to the RTS data structures which collect the coverage information. I also did not have time to look into how the default console runner is implemented, it is also important that as little testsuite driver code is executed between the initial clearTix and the final readTix that I run before and after the test, since we want to collect only the coverage of the test, not the coverage of the testsuite driver. These are the main reasons why I implement the launching logic for myself at the moment.

I am also currently working with a student on a fork of the hpc binary so that the generated html coverage report can visualize the multiple generated coverage files at the same time. We are stress-testing this with the testsuite from pandoc, and still have some performance problems with the GBs of coverage files 🙂 So I will probably iterate a while on both those tools for a while to see what exactly is required on both sides, the coverage file generation and the visualization part.

[Bodigrim]

One of the problems I foresee is that the --report-coverage function needs to disable parallel test execution.

setOption (NumThreads 1) before calling an underlying ingredient will do the trick (example).

I also did not have time to look into how the default console runner is implemented, it is also important that as little testsuite driver code is executed between the initial clearTix and the final readTix that I run before and after the test, since we want to collect only the coverage of the test, not the coverage of the testsuite driver.

My (uneducated!) guess would be to filter out any modules starting with Test.Tasty from Tix [TixModule], which does not sound too bad.

These are the main reasons why I implement the launching logic for myself at the moment.

The issue is that ignoring foldResource and foldAfter means that tasty-coverage is not executing what was supposed to be executed.

[BinderDavid]

Thanks a lot for the hints! I will try to fix the current standalone ingredient and also try out the approach with the ingredient transformer that you outlined.

My (uneducated!) guess would be to filter out any modules starting with Test.Tasty from Tix [TixModule], which does not sound too bad.

That could work, but we would also have to check that the code executed in the withResources is not attributed to only one of the tests.

The issue is that ignoring foldResource and foldAfter means that tasty-coverage is not executing what was supposed to be executed.

I have opened issue #4 for this, thanks 👍