Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add F# code coverage analysis #3756

Closed

Conversation

StachuDotNet
Copy link
Member

@StachuDotNet StachuDotNet commented Apr 30, 2022

This is a work in progress aimed to add code coverage to the F# backend. It will resolve #3688.

AltCover was chosen as the coverage tool.

Seemingly, all .NET code coverage tools assume that dotnet test will run your tests.
Expecto is not build to expose tests to dotnet test, either plainly or via [<Tests>] attributes common in .NET, so we needed to tackle this first. The NuGet packages YoloDev.Expecto.TestSdk and Microsoft.NET.Test.Sdk were added for this effect. After adding those, we also needed 1+ [<Tests>] attributes for the test-runner to discover - I opted to add this to the root Tests module containing all tests, rather than drop the attribute in many files. [<Tests>] seems to only be respected when attributing a Test binding, so Tests.fs had to adjust a bit, in order to expose such in a new tests binding.

With these changes, we can now dotnet test within the fsharp-backend directory, and it runs many of the tests well.

With dotnet test now discovering and running those tests successfully, the next step was to involve a code coverage tool - AltCover. Altcover is installed as a global tool via the dotnet tool CLI, installed in the dockerfile.

With altcover installed, we can run dotnet test /p:AltCover=true.
It seems to work at least at a basic level - here are some results:

A total of 773,417,102 visits recorded
Coverage statistics flushing took 6.862 seconds
Visited Classes 1807 of 15372 (11.76)
Visited Methods 3221 of 38990 (8.26)
Visited Points 10094 of 157756 (6.4)
Visited Branches 4714 of 134704 (3.5)
Maximum CRAP score 1176140

==== Alternative Results (includes all methods including those without corresponding source) ====
Alternative Visited Classes 3528 of 22131 (15.94)
Alternative Visited Methods 5981 of 59821 (10)
Alternative maximum CRAP score 1176140

Some outstanding tasks:

  • Adjust altcover settings to only cover our source code.
    Currently, altcover is likely ensuring that we're testing things like TestUtils, and potentially some external dependencies
  • Ensure all of our tests in Tests are included in the new tests binding
    Several test modules are commented out - some of them fail during dotnet test discovery, and others fail during dotnet test execution. The ones that fail due in discovery, I forget the details of now; will have to follow up. The ones that fail during execution are likely failing due to the sequenced aspects of those tests not being respected, though there may be additional reasons.
  • Determine where we want to store the coverage analysis and reporting (in the directory), and set up to do such
    currently, the coverage ends up at fsharp-backend/tests/Tests/coverage. I imagine we'd want to create a top-level coverage folder, with a sub-folder for pretty .html output
  • Review the actual report! Tidy things until it looks reasonable (testing the right things, with the right results)
  • Determine if we need the altcover NuGet package in addition to the global tool (i.e. can we remove it?)
    I added this when borrowing code from MiniScaffold, and need to follow up
  • Generate a nice (.html?) report from the coverage analysis
    I took an initial attempt with ReportGenerator, but sticking with AltCover's Visualizer may be our best move here. I haven't yet spent much time here. It may make sense to actually install the visualizer tool on the dev's host machine (outside of container) and then inspect the resultant analysis file, and use the traditional ReportGenerator for CI or other uses.
  • Determine if/how we want to hook coverage analysis into CI
  • Consider including tests from other test projects (fuzz? integration?)

@StachuDotNet StachuDotNet changed the title Start to add F# code coverage Add F# code coverage Apr 30, 2022
@StachuDotNet StachuDotNet changed the title Add F# code coverage Add F# code coverage analysis Apr 30, 2022
@StachuDotNet
Copy link
Member Author

This is stale, and I'm thinking to move to something other than AltCover if/when I revisit backend code coverage. (something more mainstream?). See #3688 for further discussion

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Run F# backend against coverage tool
1 participant