x/tools/refactor/eg: Using `./...` only matches one file even though there are many more

[connorszczepaniak-wk]

Go version

go version go1.23.3 darwin/arm64

Output of go env in your module/workspace:

GO111MODULE='on'
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/connorszczepaniak/Library/Caches/go-build'
GOENV='/Users/connorszczepaniak/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/connorszczepaniak/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/connorszczepaniak/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/Users/connorszczepaniak/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.3.darwin-arm64'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/Users/connorszczepaniak/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.3.darwin-arm64/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.23.3'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/connorszczepaniak/Library/Application Support/go/telemetry'
GCCGO='gccgo'
GOARM64='v8.0'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='1'
GOMOD='/path/to/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/14/dl88pgbs11766nqnslc5ry8h0000gs/T/go-build3037559288=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

I have a template looks approximately like this in abstract:

package mypkg

import (
	"testing"

	"github.com/myRepo/x/internal/a"
	"github.com/myRepo/x/internal/a/b"
)

func before(t testing.TB, widget *b.Widget, name string) string {
	return a.FunctionToReplace(t, widget, name, "", -1)
}

func after(t testing.TB, widget *b.Widget, name string) string {
	return a.NewFunction().SomethingElse(name).Another(t, widget)
}

I'm attempting to run eg -w -t template.go github.com/myRepo/... to replace all usages in this module.

What did you see happen?

When I run eg -w -t template.go ./... from the root of github.com/myRepo, I notice that only one usage gets replaced although there are many, many more in packages matched by this pattern. If I run eg -w -t template.go github.com/myRepo/x/internal/some/other/pkg, where this package contains a lot of call sites of the old function, then all of the matches appear and get replaced.

Maybe not relevant, but I noticed that for some specific packages that would be included in github.com/myRepo/..., there are errors that occur when running eg.

// when running eg -w -t template.go github.com/myRepo/x/internal/a/...
eg: template.go:11:45: cannot use widget (variable of type *b.Widget) as *b.Widget value in argument to a.FunctionToReplace

// when running eg -w -t template.go github.com/myRepo/x/internal/a/subpkg
eg: template.go:6:2: could not import github.com/myRepo/x/internal/a (package "github.com/myRepo/x/internal/a" not found)

What did you expect to see?

All call sites within the list of packages matched by github.com/myRepo/... to be replaced.

[connorszczepaniak-wk]

I've noticed this same issue with an even simpler template as well:

package mypkg

import (
	"github.com/myRepo/x/internal/a"
)

func before() a.SomeStruct {
	return a.SomeStruct{}
}

func after() a.SomeStruct{} {
	return a.SomeStruct{Field1: true}
}

[connorszczepaniak-wk]

Looked a bit deeper at the second case. It seems to be that the struct literal from before doesn't match the struct literal at the usages; namely the call to matchType here returns false even though the two are named types with the same package and same name.

If I dig into that a bit more, it looks like we fail in types.Identical when comparing named types here; even though their origins seem to be deep equal, they are not the same pointer (which is what gets compared here).

[dmitshur]

CC @golang/tools-team.

[connorszczepaniak-wk]

I did some more digging around on this:

I found one pattern/template combo for which passing github.com/path/to/package did perform the intended replacements, but for which github.com/path/to/package/... did not perform the intended replacements. In this case, my template was simply replacing a call to func A in package github.com/path/to/package with func B in the same package. In the codebase in question, github.com/path/to/package has a test referencing function A, but none of its child packages reference A at all.

In this case, it seems as though we end up eventually hitting this equality check on the objects, which, although the package paths and object names are identical, fails because they aren't the same pointer. Interestingly, if I add a check for identical package path + name, both inputs to eg now behave identically (the replacement happens for both as expected).

I've lightly browsed the code (just enough to be dangerous), and I wonder if there are problems caused by type-checking the template separately from loading the target packages or maybe some bug with merging type info or maybe something else entirely.

In any case, it seems like there are possibly multiple issues with using pointer comparison vs. using deep(ish) equality checking for type/object equivalence.

@alandonovan you seem to be the primary author of this tool, I wonder if you could weigh in?

[dmitshur]

This issue might be the same as #10142.

[connorszczepaniak-wk]

I'm not sure, that issue mentions specifically that ... patterns aren't supported, but eg (at least nowadays) uses x/tools/packages.Load for package loading which certainly supports go list syntax.

This issue is that although many subpackages are matched by ... (and therefore loaded by eg), eg often fails to find matches within them.