cmd/vet: identify 100% reliable checks for testing

[rsc]

For #18084 we need to find some way to run just the "100% reliable" vet checks. It's okay if a "100% reliable" check flags a false positive as long as there is a clean way to rewrite the code to avoid the false positive.

go test would need some way to invoke that set, like a new flag.

[mvdan]

Out of curiosity - why not disable by default those that aren't reliable on vet directly? I assume this is what go tool vet does with -shadow:

  -shadow
        check for shadowed variables (experimental; must be set explicitly)

I think the reason why go vet isn't used regularly more often is because it's not conservative by default. A good example is the standard library itself with the whitelist.

[josharian]

Perhaps so. The standard library is not a good test case though; it is unusual in virtue of being the standard library, and that accounts for much of the whitelist.

[rsc]

I think the whitelists for the standard library could be reduced dramatically by making various checks more precise or off by default.

When I spoke to @robpike about this, he believed strongly that the "default on" set should be allowed to include reports that were only, say, 90% reliable. It's hard to evaluate in the abstract but that sounds plausible.

Probably the way forward is to work on reducing false positives and figuring out the "clean rewrites" (i.e. not a //vet:goaway comment) that apply. If we can get the current default set to all be "100% reliable", that's great. If not, we'll have a concrete check to talk about instead of hypotheticals.

[btracey]

A common vet complaint in my packages is "composite literal uses unkeyed fields". If #18084 is accepted, would this be (effectively) considered a bug? One could argue that a clean rewrite is available (add the fields), but there are some good reasons not to use fields, legibility being one of them.

[rsc]

Yes, code is expected to use composite literals with keyed fields whenever the literal is initializing a struct defined in another package. Otherwise the package cannot safely add a new field without breaking clients of that package. This puts some teeth behind the requirement listed in https://golang.org/doc/go1compat. Please file a separate issue if you think we should relax this, and explain under what circumstances that make sense that would handle your case. (For example I could imagine relaxing when the package being used is an internal package.)

[calmh]

Are types from other packages that are really slices always recognized nowadays? I recall lots of issues with that warning on things like

package foo

type Vector []int

package bar

import "foo"

var v = foo.Vector{1, 2, 3}

At the very least it seemed unreliable, in that it depended on having a compiled version of foo available when vetting bar in order to avoid the warning, or something similar?

[josharian]

@calmh related issues: #16086, #11394, #15408.

[gopherbot]

Change https://golang.org/cl/74356 mentions this issue: cmd/go: run vet automatically during go test

[rogpeppe]

Another example (reported in #22715):

An example from our code base; we have an errorf method defined:

func (i *idM) errorf(w http.ResponseWriter, status int, format string, a ...interface{})

and some code is calling it:

i.errorf(w, http.StatusNotFound, "not found", "%s not found", req.URL.Path)

This produces an error from vet: errorf call has arguments but no formatting directives.
The vet -printfuncs flag seems to have lost the argument position of the first argument involved in the print, and anyway it would be better not to have to individually configure printf-like functions to get Go code to compile.

[mibk]

@rogpeppe Is the code calling it right? It seems to me that "not found" is used as the formatting parameter.

[rogpeppe]

@mibk Yes, the code is calling it right. The first "not found" is an error code, not the error description which what the format applies to.

[nathany]

I've seen the same false positive as @rogpeppe, though it was some time ago. If I recall correctly, it was from the testify Errorf function: https://godoc.org/github.com/stretchr/testify/assert#Errorf. Vet seemed to be treating it as fmt.Errorf even though the package and signature are different.

func Errorf(t TestingT, err error, msg string, args ...interface{}) bool

[rogpeppe]

@nathany To do it right, I think the format checking needs to be type sensitive, but I'm guessing that would make vet unreasonably slow.

[gopherbot]

Change https://golang.org/cl/79398 mentions this issue: all: fix test breakage due to vet checks

[gopherbot]

Change https://golang.org/cl/80596 mentions this issue: all: few tweaks to vet fixes

[ianlancetaylor]

On #23062 @calmh says

This check ["Println arg list ends with redundant newline"] is included in the new vet tests run in go test in Go 1.10 beta 1. In my opinion this is too far away from a 100% check, and mostly a style one at that. It fails tests on a binary that includes a

fmt.Println(`Usage:
    somecommand <args>

Description of the command.
More text here.
`)

and while it's correct (the string ends with a newline, and Println will add one more) this is the intended effect. You could argue that we should use cmd.Print instead and add the newline manually, and we could do that, but honestly I'm fine with it as is and think it should pass tests.

[calmh]

Seeing the initial description ("It's okay if a "100% reliable" check flags a false positive as long as there is a clean way to rewrite the code to avoid the false positive") I guess that falls under "there is a clean way to rewrite the code".

But why be so particular about the newlines? I'm skeptical about the code quality increase here.

fmt.Println("hello") // this is ok
fmt.Println("hello\n") // this is a test failure
fmt.Println("hello\n\n") // this is ok again...?

It's a trivial issue to be sure, but the inconsistency is annoying now that this is a hard failure and you start to wonder if fmt.Print("hello") should be a test failure as well because probably the programmer meant fmt.Print("hello\n") instead...

I think this test should be removed from the "100% reliable" set, both for being not 100% reliable and also for not being useful. Newline discrepancies from print are visible in the output and can be corrected accordingly by the programmer.

[rsc]

Newline discrepancies from print are visible in the output and can be corrected accordingly by the programmer.

Misused % formats are also visible in the output and can be corrected accordingly. We think they are useful because many prints are for debugging output that is not usually seen, so these bugs often linger unnoticed.

[rsc]

Also, this claims seems false:

fmt.Println("hello") // this is ok
fmt.Println("hello\n") // this is a test failure
fmt.Println("hello\n\n") // this is ok again...?

Both of the last two are reported as incorrect.

[rsc]

But why be so particular about the newlines?

If the only thing Println did was add a newline then I think it wouldn't matter much. But we made the (perhaps mistaken but locked in) decision that Println also applies a different heuristic to insert spaces around its arguments, so that fmt.Print("x", 1, "\n") might be converted to fmt.Println to get spaces around the 1, forgetting to drop the \n.

I am inclined to keep the \n check for now but I will also leave this open for a final decision later in the release cycle. /cc @robpike

[calmh]

Also, this claims seems false:

I somehow made a mistake when testing this; not sure how, but you are of course right, it's consistent.

I still think it's a silly thing to make a build error and wish we would not do this, as failing the build because someone added a newline in a non approved manner is something I'd mock a language/compiler for, but I'll concede the point if the consensus is that enforcing this is somehow desirable.

[rsc]

It's not failing the build - it's failing one test. There may be a tiny amount of initial pain here but in the steady state you only get hit by prints you just wrote.

[calmh]

No, it's not failing one test. It fails before it starts running any tests at all. While I can still compile my code without tests, exiting with a failure status before any tests are run is a build failure in my book - logically one more serious than any of the tests themselves could be. Which seems totally disproportionate for this thing. Which is why it should not be done. But I feel that I'm not saying anything you don't already know and that this is a matter of opinion, so I'm exiting here.

[rsc]

@nathany, @rogpeppe, others, the printf-check precision problem is #22936 and CLs are forthcoming.

[rsc]

Going to close this as done for Go 1.10. Hopefully in future releases we will expand the set of checks enabled during tests, but that doesn't need to be tracked in this issue.

[gopherbot]

Change https://golang.org/cl/107077 mentions this issue: cmd/go: document which vet tests are enabled during go test