Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A method to measure code coverage across tests #1267

Merged
merged 2 commits into from Jun 7, 2018
Merged

A method to measure code coverage across tests #1267

merged 2 commits into from Jun 7, 2018

Conversation

@mislav
Copy link
Member

mislav commented Sep 13, 2016

Go has code coverage tooling for test mode, which temporarily rewrites the source code to insert annotations which will activate during the test run and track progress of executed code. Then, upon process completion, that information is dumped into a coverage report.

We can't use this approach for hub, at least not without substantial changes. First of all, hub's test coverage is mostly "from the outside", utilizing Cucumber to invoke the binary with different arguments and inspect the outputs and result. There are some tests in go, but they are minimal compared to the cukes.

Second, hub frequently aborts the process on errors via os.Exit(1), and those scenarios need to be tested too. However, if the process exits prematurely, the code coverage report will never be generated.

Solution

To work around this, I first used the go tool that annotates the source:

go tool cover -mode=set -var=LiveCoverage myfile.go

This injects LiveCoverage.Count[pos] = 1 lines at appropriate places all over the source code, and generates a mapping of line/column positions in the original source.

Then I rewrite those lines to become a method invocation:

coverage.Record(LiveCoverage, pos)

The new Record method will immediately append the information to a code coverage report file as soon as it's invoked. This ensures that there is coverage information at runtime even if the process gets aborted.

This approach works the same for go tests as well as for cukes. They all append to the same file. The data in the file now resembles the format of the go coverage report:

# the columns are {file}:{startline}.{startcol},{endline}.{endcol} {num-statements} {call-count}
github.com/github/hub/cmd/cmd.go:107.31,109.4 1 0
github.com/github/hub/cmd/cmd.go:88.27,94.31 5 1
github.com/github/hub/utils/utils.go:17.23,18.16 1 0

This data needs to be massaged a little to accumulate all call counts for the same position into a single number, lest go reporting tools gets confused in the next step.

Finally, the rest of Go tooling is used to generate an HTML report of code coverage:

go tool cover -func=cover.out
go tool cover -html=cover.out

Results

$ tail tmp/cover.func

github.com/github/hub/utils/utils.go:17:                Check                           100.0%
github.com/github/hub/utils/utils.go:24:                ConcatPaths                     100.0%
github.com/github/hub/utils/utils.go:28:                BrowserLauncher                 66.7%
github.com/github/hub/utils/utils.go:41:                searchBrowserLauncher           40.0%
github.com/github/hub/utils/utils.go:62:                CommandPath                     66.7%
github.com/github/hub/utils/utils.go:80:                IsOption                        100.0%
github.com/github/hub/utils/utils.go:90:                NewColor                        70.0%
github.com/github/hub/utils/utils.go:111:               Brightness                      100.0%
github.com/github/hub/utils/utils.go:115:               TimeAgo                         64.3%
total:                                                  (statements)                    85.5%

screen shot 2016-09-13 at 1 57 12 pm

@mislav mislav changed the title Devise an evil way to measure code coverage across tests A method to measure code coverage across tests Sep 13, 2016
@xproax
xproax approved these changes Jan 16, 2017
Go has code coverage tooling for test mode, which temporarily rewrites
the source code to insert annotations which will activate during the
test run and track progress of executed code. Then, upon process
completion, that information is dumped into a coverage report.

We can't use this approach for hub, at least not without substantial
changes. First of all, hub's test coverage is mostly "from the outside",
utilizing Cucumber to invoke the binary with different arguments and
inspect the outputs and result. There are some tests in go, but they are
minimal compared to the cukes.

Second, hub frequently aborts the process on errors via `os.Exit(1)`,
and those scenarios need to be tested too. However, if the process exits
prematurely, the code coverage report will never be generated.

To work around this, I first used the go tool that annotates the source:

    go tool cover -mode=set -var=LiveCoverage myfile.go

This injects `LiveCoverage.Count[pos] = 1` lines at appropriate places
all over the source code, and generates a mapping of line/column
positions in the original source.

Then I rewrite those lines to become a method invocation:

    coverage.Record(LiveCoverage, pos)

The new `Record` method will immediately append the information to a
code coverage report file as soon as it's invoked. This ensures that
there is coverage information even if the process gets aborted.

This approach works the same for go tests as well as for cukes. They all
append to the same file. Finally, the rest of Go tooling is used to
generate an HTML report of code coverage:

    go tool cover -html=cover.out
@mislav mislav force-pushed the code-coverage branch 2 times, most recently from d50df87 to 98ebc4c Jun 7, 2018
@mislav mislav force-pushed the code-coverage branch from 98ebc4c to 429ea7d Jun 7, 2018
@mislav mislav merged commit 28104af into master Jun 7, 2018
2 checks passed
2 checks passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
@mislav mislav deleted the code-coverage branch Jun 9, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.