cmd/go: expose InvalidGoFiles as part of `go list` API

[stamblerre]

What version of Go are you using (go version)?

$ go version
go version devel +a4ba411b19 Wed Jul 1 15:29:29 2020 +0000 darwin/amd64

Does this issue reproduce with the latest release?

Yes

Repro steps

Consider a directory structure like the following:

mod2
├── go.mod
├── go.sum
├── inner
│   └── inner.go
└── main.go

1 directory, 4 files

If inner.go and main.go are both empty files, go list returns the following:

$ go list -e -json -compiled -test ./...
{
	"Dir": "/Users/rstambler/modules/mod2",
	"ImportPath": "mod2",
	"Root": "/Users/rstambler/modules/mod2",
	"Match": [
		"./..."
	],
	"Incomplete": true,
	"Error": {
		"ImportStack": [],
		"Pos": "main.go:1:1",
		"Err": "expected 'package', found 'EOF'"
	}
}
{
	"Dir": "/Users/rstambler/modules/mod2/inner",
	"ImportPath": "mod2/inner",
	"Root": "/Users/rstambler/modules/mod2",
	"Match": [
		"./..."
	],
	"Incomplete": true,
	"Error": {
		"ImportStack": [],
		"Pos": "inner/inner.go:1:1",
		"Err": "expected 'package', found 'EOF'"
	}
}

To extract the Go files associated with these packages, go/packages will have to parse the error messages.
Is there any possibility of listing the empty files in the GoFiles field?

/cc @jayconrod @bcmills @matloob

[gopherbot]

Change https://golang.org/cl/240098 mentions this issue: internal/lsp, go/packages: reproduce and fix golang/go#39646

[jayconrod]

I think we could do this. go/build.Context.ImportDir a package with these files InvalidGoFiles, along with an error. The go command sees that error and ignores the returned package. If the package is non-nil though, we could copy a few fields like this where it makes sense.

[gopherbot]

Change https://golang.org/cl/241577 mentions this issue: go/build: include files with parse errors in GoFiles and other lists

[stamblerre]

We've encountered some limitations with these CLs - specifically with this corner case:

hello.go and hello_test.go are both empty on disk, but in overlays, their contents are package hello and package hello_test. In this case, go list will report both of these files as part of hello. The result of this is that a go/packages contains query will report hello_test's package as the test variant of hello, not as an x test. This results in a lot of complexity for go/packages to work-around, and the goal of this was to avoid such work-arounds, so it doesn't seem worth it.

As an alternative, we proposed exposing the InvalidFiles list from go/build as part of the go list API. This would allow go/packages to treat the invalid files separately from a package, and then create the correct packages based on the overlays.

/cc @matloob

[gopherbot]

Change https://golang.org/cl/244028 mentions this issue: go/packages: find filenames in error strings if not in position