x/tools/gopls: workspaces ignore vendor directory when not go-getable

[zekroTJA]

gopls version

Build info
----------
golang.org/x/tools/gopls v0.8.3
    golang.org/x/tools/gopls@v0.8.3 h1:Mxm94ix8oSARQ6svioO6SxKEYWT/VCP54/448LOHzrk=
    github.com/BurntSushi/toml@v1.0.0 h1:dtDWrepsVPfW9H/4y7dDgFc2MBUSeJhlaDtK13CxFlU=
    github.com/google/go-cmp@v0.5.7 h1:81/ik6ipDQS2aGcBfIN5dHDB36BwrStyeAQquSYCV4o=
    github.com/sergi/go-diff@v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=
    golang.org/x/exp/typeparams@v0.0.0-20220218215828-6cf2b201936e h1:qyrTQ++p1afMkO4DPEeLGq/3oTsdlvdH4vqZUBWzUKM=
    golang.org/x/mod@v0.6.0-dev.0.20220106191415-9b9b3d81d5e3 h1:kQgndtyPBW/JIYERgdxfwMYh3AVStj88WQTlNDi2a+o=
    golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
    golang.org/x/sys@v0.0.0-20220209214540-3681064d5158 h1:rm+CHSpPEEW2IsXUib1ThaHIjuBVZjxNgSKmBLFfD4c=
    golang.org/x/text@v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
    golang.org/x/tools@v0.1.11-0.20220407163324-91bcfb1bdf9c h1:rYiuK/jn6TCxjAq1hQ6MR6pgOeP3A5gIHDxgrQ8Vbys=
    golang.org/x/vuln@v0.0.0-20220324005316-18fd808f5c7f h1:9dMzk88fnONra7zrEalqkRMGa9jMGf9B5mdzhYVyI28=
    golang.org/x/xerrors@v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
    honnef.co/go/tools@v0.3.0 h1:2LdYUZ7CIxnYgskbUZfY7FPggmqnh6shBqfWa8Tn3XU=
    mvdan.cc/gofumpt@v0.3.0 h1:kTojdZo9AcEYbQYhGuLf/zszYthRdhDNDUi2JKTxas4=
    mvdan.cc/xurls/v2@v2.4.0 h1:tzxjVAj+wSBmDcF6zBB7/myTy3gX9xvi8Tyr28AuQgc=
go: go1.18

go env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/rh/.cache/go-build"
GOENV="/home/rh/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/rh/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/rh/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/rh/sdk/go1.18"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/rh/sdk/go1.18/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.15"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1617301297=/tmp/go-build -gno-record-gcc-switches"

What did you do?

We have the following project structure:

src/services/api/
  cmd/
    api/
      main.go
  [other pakcages ...]
  vendor/
    somepackage/
      ...
    modules.txt
  go.mod
go.work

We have some packages in our vendor/ directory which are not go-getable, so we manage them manually in the vendor/ directory as well as in the go.mod. When the apiproject files are in the root directory, this absolutely works fine with gopls. But when using workspaces, gopls does not look up the impoted packages in the vendor directory and tries to go-get them instead, which is not possible for some packages. This results in an BrokenImport error even though go build` runs without any complaints.

What did you expect to see?

In my opinion, when the go toolchain allows the described procedure, gopls should also be able to accept packages which are not go-getable located only in the vendor/ directory.

What did you see instead?

gopls ignores the packages located in the modules vendor/ directory and tries to go-get them instead. This fails and results in a BrokenImport error even though go build runs without any problems.

Editor and settings

I've tried the following project settings for a possible chance that they could fix the problem.

{
  "gopls": {
    "build.experimentalWorkspaceModule": true,
    "build.expandWorkspaceToModule": true
  }
}

Logs

The logs shown below are not from the project described above because I don't want to leak any information about our project. But I've reconstructed the issue in a test project which shows the exact same behaviour.

[Info  - 10:18:56 AM] 2022/04/27 10:18:56 go env for /home/rh/dev/gomodtest/test
(root /home/rh/dev/gomodtest/test)
(go version go version go1.18 linux/amd64)
(valid build configuration = true)
(build flags: [])
GOINSECURE=
GOCACHE=/home/rh/.cache/go-build
GOFLAGS=
GO111MODULE=
GOPATH=/home/rh/go
GOROOT=/home/rh/sdk/go1.18
GONOPROXY=
GOPROXY=https://proxy.golang.org,direct
GOMODCACHE=/home/rh/go/pkg/mod
GOMOD=/dev/null
GONOSUMDB=
GOSUMDB=sum.golang.org
GOPRIVATE=
GOWORK=/home/rh/dev/gomodtest/test/go.work


[Info  - 10:18:59 AM] 2022/04/27 10:18:59 go/packages.Load
	snapshot=0
	directory=/home/rh/dev/gomodtest/test
	query=[builtin test/...]
	packages=3

[Error - 10:18:59 AM] 2022/04/27 10:18:59 test: no dep handle for github.com/rs/xid: no metadata for github.com/rs/xid
	snapshot=0

[Error - 10:18:59 AM] 2022/04/27 10:18:59 test: no dep handle for github.com/rs/xid: no metadata for github.com/rs/xid
	snapshot=2

[Error - 10:18:59 AM] 2022/04/27 10:18:59 test: no dep handle for github.com/rs/xid: no metadata for github.com/rs/xid
	snapshot=2

[Info  - 10:18:59 AM] 2022/04/27 10:18:59 falling back to safe trimming due to type errors: [/home/rh/sdk/go1.18/src/runtime/vdso_linux.go:53:38: invalid operation: division by zero /home/rh/sdk/go1.18/src/runtime/vdso_linux.go:54:38: invalid operation: division by zero] or still-missing identifiers: map[memRecordCycle:true pageBits:true]
	package="runtime"

[Error - 10:18:59 AM] 2022/04/27 10:18:59 tidy: diagnosing file:///home/rh/dev/gomodtest/test/test/go.mod: err: exit status 1: stderr: go: downloading github.com/rs/xid v1.5.0
test imports
	github.com/rs/xid: module lookup disabled by GOPROXY=off


[Error - 10:18:59 AM] 2022/04/27 10:18:59 tidy: diagnosing file:///home/rh/dev/gomodtest/test/test/go.mod: err: exit status 1: stderr: go: downloading github.com/rs/xid v1.5.0
test imports
	github.com/rs/xid: module lookup disabled by GOPROXY=off

[bcmills]

In general, when you combine two modules in a workspace the selected versions of the modules that provide the packages they import may be higher than the versions selected in the individual modules.

(One module in the workspace can upgrade an otherwise-irrelevant dependency vendored at a lower version in a another module, in which case the package at the correct version won't be found in any of the vendor directories at all.)

[bcmills]

This is expected in the current design of workspaces and not feasible to change.

Unfortunately, workspaces that depend on private repos need to be able to fetch those repos into the module cache.

[hyangah]

Agree that combining vendor info from multiple modules from a workspace is infeasible.

Interestingly, however, when there is only one module in a workspace, I see go list seems to work (I think) but gopls is unhappy.

Here is a case that is confusing to me.

--- go.work ---
go 1.18

use ./proj

--- proj/go.mod --
module proj

go 1.18

require golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3

require golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect

-- proj/main.go --
package main

import _ "golang.org/x/mod/modfile"

func main() {}

With the above content, prepare the following structure and run GOWORK=off go mod vendor from proj directory.

go.work
proj/
   go.mod
   go.sum
   main.go
   vendor/
      ...

Then,

$ export GOMODCACHE=`$(mktemp -d)`
$ export GOPROXY=off
$ go list -f "{{.Dir}}" golang.org/x/mod/modfile
/tmp/repro/proj/vendor/golang.org/x/mod/modfile
$ cd proj; go list -f "{{.Dir}}" golang.org/x/mod/modfile
/tmp/repro/proj/vendor/golang.org/x/mod/modfile

On the other hand, gopls fails to locate the dependency.

could not import golang.org/x/mod/modfile (no required module provides package "golang.org/x/mod/modfile") compiler [BrokenImport]

I cannot explain the difference in go and gopls's behavior.
@findleyr

[findleyr]

Reopening to investigate if we can mitigate this (partially) from gopls.