CodeCov Test Coverage Integration
CodeCov visualises code coverage and can enforce standards via GitHub and AppVeyor. More information is available on CodeCov.io.
This post is written from the point of view of someone (me) who is already proficient in C# and unit testing, but was new to AppVeyor and integrating CodeCov with GitHub.
Create a new GitHub Repo
If you are new to GitHub, see this getting started guide, otherwise simply create a new repo (YourRepositoryName from now on) and git clone it somewhere convenient.
Create a new System Under Test (SUT) project and a new Test project
In my version of Visual Studio (Community 2015), you can do this by clicking on "File - New - Project" on the main menu, then "Class Library" from "Templates - Visual C#". Give it a interesting name, which I will assume to be YourSUTProjectName for the rest of this post.
Repeat this for the test project, which I will assume to be YourTestProjectName.
xunit.runner.console nuget packages to your test project.
Add an XUnit test to YourTestProjectName and the corresponding thing under test to YourSUTProjectName, or copy and paste mine.
Run this test to make sure it passes.
Setup AppVeyor Integration
You will need to link an AppVeyor account to your GitHub one, so let's do that:
- Navigate to your repo in GitHub
- Click "Settings" on the repo
- Click "Integrations and services"
- Click "Browse Directory"
- Click "AppVeyor"
- Click "Configure"
- Click "Grant Access"
Now Log in to AppVeyor.com, probably using your GitHub account
- Click "Projects"
- Click "New Project"
- Choose YourRepositoryName and click "Add"
Sign up with CodeCov
- Go to https://codecov.io/ and click "Sign up with GitHub"
- Click "Add new repository" and choose YourRepositoryName.
- Make a note of the Upload token (YourUploadToken from now on)
Run Coverage Analysis Locally
When working with AppVeyor, it always makes sense to test on your own computer first. The feedback is immediate and you iterate very quickly. It takes a lot longer to modify the appveyor.yml file, push it and wait for a build to go through. Also, if it works locally but doesn't work on AppVeyor, you know the problem is a configuration difference between your computer and the AppVeyor environment (eg a different version of msbuild).
However, this step is not required, so skip to "Run Coverage Analysis on AppVeyor" if you wish.
CodeCov.io is a tool for visualising and integrating coverage data, which we need to create. We will use OpenCover to do this.
OpenCover nuget package to the solution (which will install OpenCover.Console.exe, probably in packages\OpenCover.4.6.519\tools) and then run the following in a command window.
- YourTestDLL is the relative path to the dll for your test project, eg. Tests\bin\Debug\tests.dll
- -register[:user] asks OpenCover to register the code coverage profiler under HKEY_CURRENT_USER, which doesn't require the user to have administrative permissions.
- -target: asks OpenCover to run XUnit
- -targetargs: are arguments that OpenCover passes to XUnit
- -output: is the file to contain the coverage results
- YourSUTNamepace, YourTestNamespace are the namespaces for your Test and SystemUnderTest projects. If these filters aren't specified OpenCover will try and analyse everything, including the XUnit dlls.
packages\OpenCover.4.6.519\tools\OpenCover.Console.exe -register:user -target:"packages\xunit.runner.console.2.2.0\tools\xunit.console.x86.exe" -targetargs:"YourTestDLL -noshadow" -output:".\coverage.xml" -filter:"+[YourSUTNamepace*]* -[YourTestNamespace*]*"
Run the following on a PowerShell command line. This talks to CodeCov and downloads a bash script to upload the coverage.
Invoke-WebRequest -Uri 'https://codecov.io/bash' -OutFile codecov.sh
Then you can run the following in a Bash command line (You can use
Git Bash on Windows). YourUploadToken is the CodeCov token that you took a note of earlier, or available on the project settings page on CodeCov (eg
codecov.sh -f "coverage.xml" -t YourUploadToken
The output will show a url with the results (eg
Run Coverage Analysis on AppVeyor
Now that the coverage upload is working locally, we can run it on AppVeyor.
Add and commit an
appveyor.yml file to the root of the repository as below.
- YourSolution is the relative path of the .sln file to build (eg codecov-on-appveyor.sln)
- YourUploadToken is not required if you are uploading from a public repo with AppVeyor, but is included in case you are using a private repo.
- %xunit20%. XUnit is installed on the AppVeyor build environment by default, and AppVeyor provide the %xunit20% environment variable for the installation path.
before_build: - nuget restore build_script: - msbuild /verbosity:quiet "YourSolution" test_script: - .\packages\OpenCover.4.6.519\tools\OpenCover.Console.exe -register:user -target:"%xunit20%\xunit.console.x86.exe" -targetargs:"YourTestDll -noshadow" -output:"coverage.xml" -filter:"+[YourSUTNamepace*]* -[YourTestNamespace*]*" after_test: - ps: | $env:PATH = 'C:\msys64\usr\bin;' + $env:PATH Invoke-WebRequest -Uri 'https://codecov.io/bash' -OutFile codecov.sh bash codecov.sh -f "coverage.xml" -t YourUploadToken
Add Coverage Graphics to the Repository
There are various badges and graphs available. Click on your project in CodeCov, then "Settings" and "Badge" (eg
https://codecov.io/gh/YourGitHubUserName/YourRepositoryName/settings/badge) to see what's available.
Integrate CodeCov with Pull Requests
CodeCov automatically integrates with GitHub pull requests (as long as you signed up to CodeCov via your GitHub account) which you can see on this pull request.
It will show some statistics, such as the increase or decrease in coverage, and by default will fail the build if coverage decreases. You can configure this with a 'codecov.yml' file in the root of your repository.
Install the browser plugin
There is a CodeCov browser plugin, for all reputable browsers, that adds coverage when browsing GitHub.com. Code is highlighted in red / green, and a coverage percentage is shown.
It can be difficult keeping control of test coverage for a project, especially with distributed and transient team structures. The combination of GitHub, AppVeyor OpenCover and CodeCov make it very easy to visualise the coverage, and allow you to enforce standards to ensure that it improves over time.