x/tools/gopls: gopls walks entire filesystem of a go module

[slon]

gopls version

Build info
----------
golang.org/x/tools/gopls v0.8.3
    golang.org/x/tools/gopls@(devel)
    github.com/BurntSushi/toml@v1.0.0 h1:dtDWrepsVPfW9H/4y7dDgFc2MBUSeJhlaDtK13CxFlU=
    github.com/google/go-cmp@v0.5.7 h1:81/ik6ipDQS2aGcBfIN5dHDB36BwrStyeAQquSYCV4o=
    github.com/sergi/go-diff@v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=
    golang.org/x/exp/typeparams@v0.0.0-20220218215828-6cf2b201936e h1:qyrTQ++p1afMkO4DPEeLGq/3oTsdlvdH4vqZUBWzUKM=
    golang.org/x/mod@v0.6.0-dev.0.20220106191415-9b9b3d81d5e3 h1:kQgndtyPBW/JIYERgdxfwMYh3AVStj88WQTlNDi2a+o=
    golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
    golang.org/x/sys@v0.0.0-20220209214540-3681064d5158 h1:rm+CHSpPEEW2IsXUib1ThaHIjuBVZjxNgSKmBLFfD4c=
    golang.org/x/text@v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
    golang.org/x/tools@v0.1.11-0.20220407163324-91bcfb1bdf9c h1:rYiuK/jn6TCxjAq1hQ6MR6pgOeP3A5gIHDxgrQ8Vbys=
    golang.org/x/vuln@v0.0.0-20220324005316-18fd808f5c7f h1:9dMzk88fnONra7zrEalqkRMGa9jMGf9B5mdzhYVyI28=
    golang.org/x/xerrors@v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
    honnef.co/go/tools@v0.3.0 h1:2LdYUZ7CIxnYgskbUZfY7FPggmqnh6shBqfWa8Tn3XU=
    mvdan.cc/gofumpt@v0.3.0 h1:kTojdZo9AcEYbQYhGuLf/zszYthRdhDNDUi2JKTxas4=
    mvdan.cc/xurls/v2@v2.4.0 h1:tzxjVAj+wSBmDcF6zBB7/myTy3gX9xvi8Tyr28AuQgc=
go: go1.18

go env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/prime/.cache/go-build"
GOENV="/home/prime/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/prime/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/prime/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/prime/.ya/tools/v4/3102076311"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/prime/.ya/tools/v4/3102076311/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.18.2"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/prime/arc/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/var/tmp/go-build3354655002=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Our internal version control system is working over fuse. Big mono-repository is mounted as a virtual file system and loaded lazily on demand. Whole repository is a single go module with single go.mod file at the root.

When I open subset of big mono-repository in vscode, gopls starts walking all directories of the repository, breaking lazy load.

What did you expect to see?

gopls should interact only with directories added to workspace.

What did you see instead?

gopls scans all directories of a go module.

This issue originates imports cache. https://github.com/golang/tools/blob/cf66aec62b42a726b36e083c001d3b3231cba2b3/internal/imports/fix.go#L675-L689

It loads all directories from disk without any way of setting up excludes.

I tried adding exclude filter to gopls configuration. This patch fixed issue for me: slon/tools@bfcfbb1

Editor and settings

{
	"folders": [
		{
			"path": "arc/yt"
		},
		{
			"path": "arc/mapreduce"
		},
		{
			"path": "arc/library/cpp"
		},
		{
			"path": "arc/junk/prime"
		},
		{
			"path": "arc/yp"
		},
		{
			"path": "arc/devtools/yamaker"
		},
		{
			"path": "arc/devtools/ya"
		},
		{
			"path": "arc/web/app_host/api/service"
		},
		{
			"path": "arc/library"
		},
		{
			"path": "arc/contrib/libs/tcmalloc"
		}
	],
	"settings": {
        "go.toolsEnvVars": {
            "CGO_ENABLED": "0"
        }, 
        "go.logging.level": "verbose", 
        "go.goroot": "/home/prime/.ya/tools/v4/1272507071", 
        "gopls": {
            "verboseOutput": true, 
            "formatting.local": "a.yandex-team.ru", 
            "build.env": {
                "CGO_ENABLED": "0"
            }, 
            "build.expandWorkspaceToModule": false,
        }, 
        "files.watcherExclude": {
            "**": true
        }
    }
}

Logs

[findleyr]

Thanks for the issue, and for the patch. We should reuse the directoryFilters setting, or go.work files, rather than add a new setting. But otherwise what you did looks good.

Rationalizing these settings with go.work is in the plan for gopls@v0.9.0, so I'm including this in that milestone (our next release).

[slon]

directoryFilters setting is broken in my case.

My workspace consists of multiple folders. directoryFilter is evaluated relative to each directory. But import cache filter should be evaluated relative to module root.

Because of this issue, I added separate setting in my proof-of-concept patch.

It is possible to reuse directoryFilter, but this would require changing semantics in case where workspace directory is not module root.

[findleyr]

I see! Thank you for clarifying. We will fix this.

[findleyr]

Our plan to fix this is to extend directoryFilters to support wildcards, and pass this to goimports. @dle8 is working on the former feature.

https://go.dev/issue/52784 also fixed a bug in gopls that caused scanning of GOPATH when using go.work.

[gopherbot]

Change https://go.dev/cl/414454 mentions this issue: internal/lsp: Add exclude filters to gopls ModuleResolver.scan