x/tools/go/packages: overlay does not work properly with external module files

[mateusz834]

Consider:

package main

import (
	"fmt"

	"golang.org/x/tools/go/packages"
)

func main() {
	fmt.Println(run())
}

func run() error {
	cfg := packages.Config{
		Mode: packages.LoadSyntax,
		Dir:  ".",
		Overlay: map[string][]byte{
			"/home/mateusz/go/pkg/mod/golang.org/x/tools@v0.28.0/go/packages/nonExistent.go": []byte(`package packages; func TestFunc() {}`),  // New file
			"/home/mateusz/go/pkg/mod/golang.org/x/tools@v0.28.0/go/packages/doc.go":         []byte(`package packages; func TestFunc2() {}`), // File exists, but overridden.

			// File exists, but overridden, additional import (unicode/utf8).
			"/home/mateusz/go/pkg/mod/golang.org/x/tools@v0.28.0/go/packages/loadmode_string.go": []byte(`// Copyright 2019 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

package packages

import (
	"fmt"
	"strings"
	"unicode/utf8"
)

func TestFunc3(r rune) []byte {
	return utf8.AppendRune(nil, r)
}

var modes = [...]struct {
	mode LoadMode
	name string
}{
	{NeedName, "NeedName"},
	{NeedFiles, "NeedFiles"},
	{NeedCompiledGoFiles, "NeedCompiledGoFiles"},
	{NeedImports, "NeedImports"},
	{NeedDeps, "NeedDeps"},
	{NeedExportFile, "NeedExportFile"},
	{NeedTypes, "NeedTypes"},
	{NeedSyntax, "NeedSyntax"},
	{NeedTypesInfo, "NeedTypesInfo"},
	{NeedTypesSizes, "NeedTypesSizes"},
	{NeedForTest, "NeedForTest"},
	{NeedModule, "NeedModule"},
	{NeedEmbedFiles, "NeedEmbedFiles"},
	{NeedEmbedPatterns, "NeedEmbedPatterns"},
}

func (mode LoadMode) String() string {
	if mode == 0 {
		return "LoadMode(0)"
	}
	var out []string
	// named bits
	for _, item := range modes {
		if (mode & item.mode) != 0 {
			mode ^= item.mode
			out = append(out, item.name)
		}
	}
	// unnamed residue
	if mode != 0 {
		if out == nil {
			return fmt.Sprintf("LoadMode(%#x)", int(mode))
		}
		out = append(out, fmt.Sprintf("%#x", int(mode)))
	}
	if len(out) == 1 {
		return out[0]
	}
	return "(" + strings.Join(out, "|") + ")"
}
`),
		},
	}
	pkgs, err := packages.Load(&cfg, "./test")
	if err != nil {
		return err
	}
	packages.PrintErrors(pkgs)
	return nil
}

This program runs packages.Load on a ./test package, with three overlay files on the golang.org/x/tools/go/packages package.

• nonExistent.go - File does not exist, so this is a new file with a new func TestFunc.
• doc.go - File already exists, it defines a new func TestFunc2.
• loadmode_string.go - File already exists, it defines a new func TestFunc3 and imports unicode/utf8 (this package is not imported without overlays).

The ./test package contains following file:

package main

import (
	"golang.org/x/tools/go/packages"
)

func main() {
	packages.TestFunc()
	packages.TestFunc2()
	packages.TestFunc3('a')
}

$ go run .
-: # golang.org/x/tools/go/packages
/tmp/gocommand-708375563/3-loadmode_string.go:10:2: could not import unicode/utf8 (open : no such file or directory)
/home/mateusz/go/pkg/mod/golang.org/x/tools@v0.28.0/go/packages/loadmode_string.go:10:2: could not import unicode/utf8 (no metadata for unicode/utf8)
/tmp/aa/test/test.go:8:11: undefined: packages.TestFunc
<nil>

So:

• It is not able to import unicode/utf8 (probably go list does not return it to go/packages).
• Newly added files are ignored (nonExistent.go).
• Overlay works, but only for files that already exist.

Opened this issue, because i think that this behavior is wrong, it should either work without any error for the example above, or just ignore the overlay completely.

CC @matloob (per https://dev.golang.org/owners) @adonovan

[gabyhelp]

Related Issues

• x/tools/go/packages: returns wrong package name when overlay has a new package name #38328 (closed)
• cmd/go: go test does not parse -overlay files for test functions #44946 (closed)
• x/tools/go/packages: NeedExportFile causes error about invalid go.mod file to be dropped #67638
• x/tools/go/packages: passing NeedSyntax to Load() doesn't populate Syntax field unless NeedTypes is passed #35331 (closed)
• x/tools/go/packages: Load in NeedName|NeedFiles mode can produce incorrect results for certain modules #31894 (closed)
• x/tools/go/loader: tests fail with go 1.5 tip #11902 (closed)
• x/tools/go/packages: clarify naming convention of overlay files #67541
• x/tools/go/packages: NeedSyntax without NeedFiles/NeedCompiledGoFiles produces empty slice #56677
• x/tools/go/loader: broken tests #11920 (closed)

Related Code Changes

• cmd/go: support overlaying go.mod files

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[mateusz834]

This error comes directly from go list, and the nonExistent.go is not returned (as described in the issue).

GOROOT= GOPATH=/home/mateusz/go GO111MODULE= GOPROXY= PWD=. go list -overlay=/tmp/gocommand-417420243/overlay.json -e -json=Name,ImportPath,Error,Dir,GoFiles,IgnoredGoFiles,IgnoredOtherFiles,CFiles,CgoFiles,CXXFiles,MFiles,HFiles,FFiles,SFiles,SwigFiles,SwigCXXFiles,SysoFiles,CompiledGoFiles,Export,DepOnly,Imports,ImportMap,Module -compiled=true -test=false -export=true -deps=true -find=false -pgo=off -- .

{
        "Dir": "/home/mateusz/go/pkg/mod/golang.org/x/tools@v0.28.0/go/packages",
        "ImportPath": "golang.org/x/tools/go/packages",
        "Name": "packages",
        "Module": {
                "Path": "golang.org/x/tools",
                "Version": "v0.28.0",
                "Time": "2024-12-04T22:20:41Z",
                "Dir": "/home/mateusz/go/pkg/mod/golang.org/x/tools@v0.28.0",
                "GoMod": "/home/mateusz/go/pkg/mod/cache/download/golang.org/x/tools/@v/v0.28.0.mod",
                "GoVersion": "1.22.0",
                "Sum": "h1:WuB6qZ4RPCQo5aP3WdKZS7i595EdWqWR8vqJTlwTVK8=",
                "GoModSum": "h1:dcIOrVd3mfQKTgrDVQHqCPMWy6lnhfhtX3hLXYVLfRw="
        },
        "DepOnly": true,
        "GoFiles": [
                "doc.go",
                "external.go",
                "golist.go",
                "golist_overlay.go",
                "loadmode_string.go",
                "packages.go",
                "visit.go"
        ],
        "CompiledGoFiles": [
                "doc.go",
                "external.go",
                "golist.go",
                "golist_overlay.go",
                "loadmode_string.go",
                "packages.go",
                "visit.go"
        ],
        "Imports": [
                "bytes",
                "context",
                "encoding/json",
                "errors",
                "fmt",
                "go/ast",
                "go/parser",
                "go/scanner",
                "go/token",
                "go/types",
                "golang.org/x/sync/errgroup",
                "golang.org/x/tools/go/gcexportdata",
                "golang.org/x/tools/internal/gocommand",
                "golang.org/x/tools/internal/packagesinternal",
                "golang.org/x/tools/internal/typesinternal",
                "encoding/json",
                "errors",
                "fmt",
                "go/ast",
                "go/parser",
                "go/scanner",
                "go/token",
                "go/types",
                "golang.org/x/sync/errgroup",
                "golang.org/x/tools/go/gcexportdata",
                "golang.org/x/tools/internal/gocommand",
                "golang.org/x/tools/internal/packagesinternal",
                "golang.org/x/tools/internal/typesinternal",
                "log",
                "os",
                "os/exec",
                "path",
                "path/filepath",
                "reflect",
                "runtime",
                "slices",
                "sort",
                "strconv",
                "strings",
                "sync",
                "sync/atomic",
                "time",
                "unicode"
        ],
        "Error": {
                "ImportStack": null,
                "Pos": "",
                "Err": "# golang.org/x/tools/go/packages\n/tmp/gocommand-417420243/3-loadmode_string.go:10:2: could not import unicode/utf8 (open : no such file or directory)\n"
        }
}

[mateusz834]

Doing the same thing, but with the x/tools module under go.work (so that it does not use the mod cache):

GOROOT= GOPATH=/home/mateusz/go GO111MODULE= GOPROXY= PWD=. go list -overlay=/tmp/gocommand-2002914138/overlay.json -e -json=Name,ImportPath,Error,Dir,GoFiles,IgnoredGoFiles,IgnoredOtherFiles,CFiles,CgoFiles,CXXFiles,MFiles,HFiles,FFiles,SFiles,SwigFiles,SwigCXXFiles,SysoFiles,CompiledGoFiles,Export,DepOnly,Imports,ImportMap,Module -compiled=true -test=false -export=true -deps=true -find=false -pgo=off -- .

{
	"Dir": "/tmp/aa/tools/go/packages",
	"ImportPath": "golang.org/x/tools/go/packages",
	"Name": "packages",
	"Export": "/home/mateusz/.cache/go-build/66/66d1428ddfbc9ee5f6e0c00f1827659c07e965a523f71ce88981612ced939ddd-d",
	"Module": {
		"Path": "golang.org/x/tools",
		"Main": true,
		"Dir": "/tmp/aa/tools",
		"GoMod": "/tmp/aa/tools/go.mod",
		"GoVersion": "1.22.0"
	},
	"DepOnly": true,
	"GoFiles": [
		"doc.go",
		"external.go",
		"golist.go",
		"golist_overlay.go",
		"loadmode_string.go",
		"nonExistent.go",
		"packages.go",
		"visit.go"
	],
	"CompiledGoFiles": [
		"doc.go",
		"external.go",
		"golist.go",
		"golist_overlay.go",
		"loadmode_string.go",
		"nonExistent.go",
		"packages.go",
		"visit.go"
	],
	"Imports": [
		"bytes",
		"context",
		"encoding/json",
		"errors",
		"fmt",
		"go/ast",
		"go/parser",
		"go/scanner",
		"go/token",
		"go/types",
		"golang.org/x/sync/errgroup",
		"golang.org/x/tools/go/gcexportdata",
		"golang.org/x/tools/internal/gocommand",
		"golang.org/x/tools/internal/packagesinternal",
		"golang.org/x/tools/internal/typesinternal",
		"log",
		"os",
		"os/exec",
		"path",
		"path/filepath",
		"reflect",
		"runtime",
		"slices",
		"sort",
		"strconv",
		"strings",
		"sync",
		"sync/atomic",
		"time",
		"unicode",
		"unicode/utf8"
	]
}

[matloob]

Hi, I'd like to understand your use case better, but in general this is not supported.

We expect the module cache to be immutable (and we write all the files and directories in the cache readonly), and I think it would be impractical to change that expectation.

One thing we could do is to detect if files were overlaid in the module cache and explicitly reject that at the beginning. We could also do the same, with, for example the build cache.

[mateusz834]

Hi, I'd like to understand your use case better, but in general this is not supported.

I don't have any particular use case, i just mistakenly overridden the module cache files (in the overlay) and spotted this behavior. I am fine if it didn't worked at all. The current behavior and the error message open : no such file or directory (no file name) made me think that this is a bug.

I am not entirely sure what did you mean by "and explicitly reject that at the beginning"? Load (go/packages) returning an error or (as my suggestion) ignoring the overlay files for such cases?

[matloob]

I meant to return an error. I'm thinking of things from the go command perspective: go/packages invokes go list to get the information it needs. I'm proposing that the go command returns an error, which is in turn is returned by go/packages.

[mateusz834]

I meant to return an error. I'm thinking of things from the go command perspective: go/packages invokes go list to get the information it needs. I'm proposing that the go command returns an error, which is in turn is returned by go/packages.

Directly by Load or in Package.Error? I think it should be in Package.Error since otherwise it might break LSP (people can accidentally edit module cache files).

[matloob]

cc @findleyr

Oh, that's a good point. I was thinking directly by Load, but you're right that that would cause problems. It would be more complicated to figure out how to set the error on the package from cmd/go.

Alternatively we can maybe have gopls check for overlay files in the modcache before calling packages.Load and filtering those out.