proposal: testing: add magic //go:cover ignore comment

[earthboundkid]

Problem

The Go coverage tool is very useful, but there's no way to mark a line as not needing coverage. It is common to add panic("unreachable") in place where one expects code never to run unless there is a programmatic error. These lines show up in code coverage output as untested and must be manually inspected in order to be noted that they can be ignored and don't need to be tested.

Proposal

If a line is prefaced by //go:cover ignore the following expression or block will be marked as "ignored".

// some switch ...
case 1:
  return "1"
//go:cover ignore
default: // this line and the next will be marked as ignored
   panic("unreachable")
// ...

$ go test -coverprofile=coverage.out
PASS
coverage: 42% of statements (2% ignored)
ok      size    0.030s

In the HTML output of the cover tool, ignored lines should be marked with yellow. If an ignored line is covered during a test, it should be reported as "ignored but covered" and marked in purple for investigation of whether a programmatic error has taken place.

Cf #31280

[seankhliao]

same as #34639

[randall77]

See also #12504

[earthboundkid]

Based on prior comments, Rob Pike is opposed to magic comments to control the cover tool. That makes sense, but if there's no knob for controlling the cover tool, it would be good if the tool had some way of detecting and ignoring panic("unreachable") lines on its own. Maybe panic("unreachable") should be its own magic comment?

[Jorropo]

Maybe panic("unreachable") should be its own magic comment?

The discoverability of that feature for new users sounds really low.

I think, a builtin (or somewhere in the std):

func unreachable() {
  panic("unreachable")
}

And the magic comment being unreachable() is better.

This is a feature other modern languages integrates, for example see in rust or GCC & Clang extensions.

[earthboundkid]

See #30582 for an unreachable in the standard library.

[breml]

I understand, that Rob Pike has a strong voice, but what is about the community?

As mentioned in #31280 (comment), a substantial part of the community already adopted tools, that work with magic comments to handle false positives. The //go:cover ignore comment would do a similar thing and I would not be surprised, if the same parts of the community would adopt this feature.

If such a magic comment is introduced, nobody is required to use it and the Go team can forbid to use it in the Go standard library, the Go tooling, etc. if they really wish to do this.

[ianlancetaylor]

If we're never going to use them in the Go standard library and tooling, then it doesn't seem too hard to me to wrap the Go coverage tool with a wrapper that looks for the comments itself.

If we add direct support in the Go coverage tool, we will be regularly rejecting patches to add them to the Go standard library.

[rsc]

I'm not sure why this has to be part of cover. If you have something that reports the uncovered lines, couldn't it filter them out instead? In general we're thinking about redoing the way cover works to be more inside the compiler, and I'd rather not have the compiler need to introduce per-basic-block magic comments as a concept.

[rsc]

I wrote this tool a while back: https://pkg.go.dev/rsc.io/tmp/uncover.
You could imagine doing something similar and filtering in that tool instead.
The only time this would need to be in cover is to affect the reported percentage.
Maybe we should remove the reported percentages because people focus too much on them?

[Jorropo]

See #30582 for an unreachable in the standard library.

This seems to be a different proposal.
It seems to care about performance and this would be a compiler hint, with an unsafe mode that let the compiler prune unreachable branches.

What I propose is functionally equivalement to panic("unreachable") except the coverage tool would understand it (not like the Gcc builtin then, I've removed that part).

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rsc]

What I propose is functionally equivalement to panic("unreachable") except the coverage tool would understand it

Why is it important for the coverage tool to "understand it"?
Are people focusing too much on the percentages?

[earthboundkid]

I think the conversation is becoming a bit circular. The point isn't the percentage. It's to communicate to other devs/one's future self that a certain line can be ignored when looking at coverage summaries. Otherwise one ends up opening files that don't need to be opened just to see what the 1% uncovered lines are and then immediately closing it again after deciding that yes, the panic is unreachable. It's a waste of time.

[rsc]

If that's the case, then what if cover "grays out" panic("unreachable")?
Is that sufficient?
That's a lot more palatable to me than //go:cover ignore.

[seankhliao]

In my experience, "coverage ignore" comments are used by developers as a hammer to pass coverage gates in CI (usually not 100%), excluding things they don't know how to test or otherwise don't understand why is only partially covered, or just fudging things when they're 0.5% below the gate.

If an ignored line is covered during a test, it should be reported as "ignored but covered" and marked in purple for investigation of whether a programmatic error has taken place.

That this is even possible makes this, imo, the wrong approach. After all, it's easy to just mark things uncovered and don't write test cases for them.

As someone responsible for developer tooling, ignore comments reduce my trust in the system accurately reflecting reality. I'm sure I'm not the only one having seen services crash with "unreachable" panics, these untested code paths should continue to be reported as such.

[earthboundkid]

I'm leaving this issue open, because there are still other users who support it, but I think my preference at this point would be something like #30582 being automatically marked as ignored by coverage. If something screws up and the line does get run, it will trigger a panic, so there's no need for a "purple" accidentally covered code color.

[thanm]

I'd like to understand the exact semantics of this proposed directive a bit better.

Most existing Go compiler directives (ex: go:noinline, go:noescape, etc) apply to the thing that immediately follows the directive. Is this also the case for the proposed directive? Or do we need some sort of control flow analysis to figure out the set of statements to which it applies?

For example, in this code, does the directive only apply to the statement on line 11, or also to the statement on line 9 as well? I would assume this to be the case, given that our code coverage setup doesn't take into account panic paths.

  func example1(int x) {
    doSomething()          // line 9
    //go:cover ignore      // line 10
    doAnotherThing()       // line 11
    return 42
  }

What about the function below-- if control flow reaches "return 42", this implies that the "if" statement has executed. Do we apply the "no cover" property to the "if" statement? That would be consistent with the previous example (again, assuming that we're not taking into account panic paths). What about the println in the defer statement?

func example2(int x) int {
  defer func() {
    println("himom")
  }()
  if x == 101 {
     launchThePaperAirplane()
  } else {
     orderACuppaCoffee()
  }
  //go:cover ignore
  return 42
}

Where can I place go:nocover directives? Is either of these directives illegal?

//go:nocover
func example3(int x) int {
  x++
  if x < 2 &&
  //go:nocover
  x > 9 {
     launchThePaperAirplane()
  }
  return 0
}

If there are multiple directives, should this be taken into account during control flow analysis? In the function below, both arms of the "if" statement have a //go:nocover directive. This implies that the "if" statement should also not be covered-- should the launchThePaperAirplane() call be also considered nocover?

func example4(int x) int {
  if x == 0 {
    //go:nocover
    x++
  } else {
    //go:nocover
    return x
  }
  return launchThePaperAirplane()
}

[breml]

@thanm I executed your examples with some "prior art" (courtney) with the idea to give a possible answer. But this is by no means the single correct answer, just a possible one, because the respective implementation does already exist.

example1

courtney, does mark the whole function as "not tracked". This is inline with your assumption, because possible panic paths are not considered and courtney operates on block level.

example2

courtney considers the code after the if-else block as a fresh block, which is not linked to the previous one. Therefore the // notest comment does only operate on the code block after the if-else.

example3a (at the end of the file in the screenshot)

The // notest comment before func example3a does have the effect, that from this position on in this file, every line is considered as "not tracked".

The // notest comment is not considered illegal by courtney.

example3b

The if condition is considered part of the first code block of this function and therefore x++ as well as if x < 2 && x > 9 is considered "not tracked". The reason for this is, that courtney operates on the coverage information generated by go tool cover, where as of now, this is all one block.

The // notest comment is not considered illegal by courtney.

example 4

courtney does not do a control flow analysis and therefore it shows both arms of the if-else as "not tracked" where the statements before and after the if-else (including the condition for the if) is not affected by the // notest comments.

Screenshot of the HTML cover report

The HTML cover report by go tool cover -html=coverage.out where coverage.out is generated by courtney.

Summary

courtney operates on the code blocks emitted by the existing implementation of go tool cover and does not add more details or more fine grained analysis of possible code flows. If there is a // notest comment at any position within a code block, this code block (as well as all code blocks following on the same or a more deeply nested level) are affected by the // notest comment and therefore marked as "not tracked".

[breml]

@thanm The following is my personal point of view to the examples you have provided.

In general, when I think about coverage, I think in terms of branches (or code block) and I didn't care to much about potential panic paths until now (maybe, this discussion will change this, we will see).

example1

Because I think in terms of code blocks, I expect the //go:cover ignore comment to affect all statements of this function, regardless of the exact position of the comment. That being said, me personally, I would put the //go:cover ignore comment as the first line with the function body and I would see this as a best practice.
So yes, I agree with your assumption.

example2

Form my point of view, this is a rather fabricated example. The request for //go:cover ignore comments comes from real needs with real code and real tests. Therefore I would not expect the cover tool to make any "intelligent" assumptions about control flow. So I think, courtney does the right thing here, even if it is rather meaningless. In practice, I don't expect this case to happen. Either the whole function is not covered and in this case the comment should go as first line within the function. If this is a real case, because every code path leads to a panic and we want indeed to ignore the return 42, then again, ignoring this code block is the right thing to do here.

example3

For the first //go:nocover comment, before the function declaration, I think I would like it to only affect this function (in contrast to what courtney is doing. This would be in line with e.g. how the //nolint comments from golangci-lint work. That being said, I can life with the way courtney is handling this.

The second //go:nocover comment is a little bit more trick. Currently, go tool cover does not consider binary expressions as different code paths so again, counting the comment to the first code block (x++ ... x > 9) is the right thing in my opinion. If go tool cover does learn about binary expressions, then I would expect this comment to only affect the second expression (x > 9).

example4

Here again, my personal opinion is, that go tool cover does not need to take control flow analysis into account. From my personal experience, this is not a case I expect to happen in real code.

[breml]

In my experience, "coverage ignore" comments are used by developers as a hammer to pass coverage gates in CI (usually not 100%), excluding things they don't know how to test or otherwise don't understand why is only partially covered, or just fudging things when they're 0.5% below the gate.

If this is the case in a development team, then this is a cultural issue, which can not be solved with a tool. If the option for "coverage ignore` comments does not exist, such developers find other ways to cheat the system.

As someone responsible for developer tooling, ignore comments reduce my trust in the system accurately reflecting reality. I'm sure I'm not the only one having seen services crash with "unreachable" panics, these untested code paths should continue to be reported as such.

The thing is, the panic will happen regardless of the fact if the code is pushed to production with the "real" coverage reported in CI or with an "adjusted" coverage. Where I work, we can see the coverage in our review process and the important part is the discussion, why some code is not covered and what can be done about it. After this discussion, in my opinion it is fine to let the tooling know (with a magic comment), that not covering a certain section is ok.
Think of this as an accepted risk. In risk management, not all risks can be mitigated, for some risks, companies decide to accept them. This process is also documented. For me, //go:cover ignore comments have a similar usage.

Just today I had a case, with an error path, that is so unlikely, that I am happy to accept the risk of not having a test for this case. The respective code processed a JWT token. After the signature has been validated, the claims are extracted from the token, which is basically JSON unmarshal. When I got a JWT with a valid signature, chances are very small, that it contains an invalid JSON body for the claims. I don't say it is impossible to happen nor do I say, that it is not possible to create a test for this case, but from a business point of view it is just not worth the effort to write such a test in contrast to the risk.

[rsc]

Based on the discussion above, this proposal seems like a likely decline.
— rsc for the proposal review group

[breml]

Maybe I should save my energy for other topics, because it seams, that this ship has likely sailed. I just want to reply to one more point. @rsc wrote some comments back:

I'm not sure why this has to be part of cover. If you have something that reports the uncovered lines, couldn't it filter them out instead? In general we're thinking about redoing the way cover works to be more inside the compiler, and I'd rather not have the compiler need to introduce per-basic-block magic comments as a concept.

One reason, why this needs to be part of cover (and the go tool) is editor integration. While it is possible to post process the coverage files e.g. on the command line or in CI, this is normally not an option in the editor.
I for example use VS Code, where I can just configure an alternative for the whole go tool, but I have no possibility (that I am aware of) to configure additional tools to post-process the coverage report, before it is visualized for me in VS Code. I assume, that this is also true for other editor integrations.
The thing is, the whole testing and coverage measurement tooling is tightly coupled with the go tool. As of now, editor integration with post-processed coverage information is only possible by creating a complete facade of the go tool, intercept the relevant call and maybe even using -toolexec to inject the additional processing (similar to how it is done by garble).

[rsc]

No change in consensus, so declined.
— rsc for the proposal review group