Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools/gopls: workspaces ignore vendor directory when not go-getable #52587

Open
zekroTJA opened this issue Apr 27, 2022 · 4 comments
Open

x/tools/gopls: workspaces ignore vendor directory when not go-getable #52587

zekroTJA opened this issue Apr 27, 2022 · 4 comments
Labels
gopls/workspace gopls modules NeedsInvestigation Tools Unfortunate
Milestone

Comments

@zekroTJA
Copy link

@zekroTJA zekroTJA commented Apr 27, 2022

gopls version

Build info
----------
golang.org/x/tools/gopls v0.8.3
    golang.org/x/tools/gopls@v0.8.3 h1:Mxm94ix8oSARQ6svioO6SxKEYWT/VCP54/448LOHzrk=
    github.com/BurntSushi/toml@v1.0.0 h1:dtDWrepsVPfW9H/4y7dDgFc2MBUSeJhlaDtK13CxFlU=
    github.com/google/go-cmp@v0.5.7 h1:81/ik6ipDQS2aGcBfIN5dHDB36BwrStyeAQquSYCV4o=
    github.com/sergi/go-diff@v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=
    golang.org/x/exp/typeparams@v0.0.0-20220218215828-6cf2b201936e h1:qyrTQ++p1afMkO4DPEeLGq/3oTsdlvdH4vqZUBWzUKM=
    golang.org/x/mod@v0.6.0-dev.0.20220106191415-9b9b3d81d5e3 h1:kQgndtyPBW/JIYERgdxfwMYh3AVStj88WQTlNDi2a+o=
    golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
    golang.org/x/sys@v0.0.0-20220209214540-3681064d5158 h1:rm+CHSpPEEW2IsXUib1ThaHIjuBVZjxNgSKmBLFfD4c=
    golang.org/x/text@v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
    golang.org/x/tools@v0.1.11-0.20220407163324-91bcfb1bdf9c h1:rYiuK/jn6TCxjAq1hQ6MR6pgOeP3A5gIHDxgrQ8Vbys=
    golang.org/x/vuln@v0.0.0-20220324005316-18fd808f5c7f h1:9dMzk88fnONra7zrEalqkRMGa9jMGf9B5mdzhYVyI28=
    golang.org/x/xerrors@v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
    honnef.co/go/tools@v0.3.0 h1:2LdYUZ7CIxnYgskbUZfY7FPggmqnh6shBqfWa8Tn3XU=
    mvdan.cc/gofumpt@v0.3.0 h1:kTojdZo9AcEYbQYhGuLf/zszYthRdhDNDUi2JKTxas4=
    mvdan.cc/xurls/v2@v2.4.0 h1:tzxjVAj+wSBmDcF6zBB7/myTy3gX9xvi8Tyr28AuQgc=
go: go1.18

go env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/rh/.cache/go-build"
GOENV="/home/rh/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/rh/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/rh/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/rh/sdk/go1.18"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/rh/sdk/go1.18/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.15"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1617301297=/tmp/go-build -gno-record-gcc-switches"

What did you do?

We have the following project structure:

src/services/api/
  cmd/
    api/
      main.go
  [other pakcages ...]
  vendor/
    somepackage/
      ...
    modules.txt
  go.mod
go.work

We have some packages in our vendor/ directory which are not go-getable, so we manage them manually in the vendor/ directory as well as in the go.mod. When the apiproject files are in the root directory, this absolutely works fine with gopls. But when using workspaces, gopls does not look up the impoted packages in the vendor directory and tries to go-get them instead, which is not possible for some packages. This results in an BrokenImport error even though go build` runs without any complaints.

What did you expect to see?

In my opinion, when the go toolchain allows the described procedure, gopls should also be able to accept packages which are not go-getable located only in the vendor/ directory.

What did you see instead?

gopls ignores the packages located in the modules vendor/ directory and tries to go-get them instead. This fails and results in a BrokenImport error even though go build runs without any problems.

Editor and settings

I've tried the following project settings for a possible chance that they could fix the problem.

{
  "gopls": {
    "build.experimentalWorkspaceModule": true,
    "build.expandWorkspaceToModule": true
  }
}

Logs

The logs shown below are not from the project described above because I don't want to leak any information about our project. But I've reconstructed the issue in a test project which shows the exact same behaviour.

[Info  - 10:18:56 AM] 2022/04/27 10:18:56 go env for /home/rh/dev/gomodtest/test
(root /home/rh/dev/gomodtest/test)
(go version go version go1.18 linux/amd64)
(valid build configuration = true)
(build flags: [])
GOINSECURE=
GOCACHE=/home/rh/.cache/go-build
GOFLAGS=
GO111MODULE=
GOPATH=/home/rh/go
GOROOT=/home/rh/sdk/go1.18
GONOPROXY=
GOPROXY=https://proxy.golang.org,direct
GOMODCACHE=/home/rh/go/pkg/mod
GOMOD=/dev/null
GONOSUMDB=
GOSUMDB=sum.golang.org
GOPRIVATE=
GOWORK=/home/rh/dev/gomodtest/test/go.work


[Info  - 10:18:59 AM] 2022/04/27 10:18:59 go/packages.Load
	snapshot=0
	directory=/home/rh/dev/gomodtest/test
	query=[builtin test/...]
	packages=3

[Error - 10:18:59 AM] 2022/04/27 10:18:59 test: no dep handle for github.com/rs/xid: no metadata for github.com/rs/xid
	snapshot=0

[Error - 10:18:59 AM] 2022/04/27 10:18:59 test: no dep handle for github.com/rs/xid: no metadata for github.com/rs/xid
	snapshot=2

[Error - 10:18:59 AM] 2022/04/27 10:18:59 test: no dep handle for github.com/rs/xid: no metadata for github.com/rs/xid
	snapshot=2

[Info  - 10:18:59 AM] 2022/04/27 10:18:59 falling back to safe trimming due to type errors: [/home/rh/sdk/go1.18/src/runtime/vdso_linux.go:53:38: invalid operation: division by zero /home/rh/sdk/go1.18/src/runtime/vdso_linux.go:54:38: invalid operation: division by zero] or still-missing identifiers: map[memRecordCycle:true pageBits:true]
	package="runtime"

[Error - 10:18:59 AM] 2022/04/27 10:18:59 tidy: diagnosing file:///home/rh/dev/gomodtest/test/test/go.mod: err: exit status 1: stderr: go: downloading github.com/rs/xid v1.5.0
test imports
	github.com/rs/xid: module lookup disabled by GOPROXY=off


[Error - 10:18:59 AM] 2022/04/27 10:18:59 tidy: diagnosing file:///home/rh/dev/gomodtest/test/test/go.mod: err: exit status 1: stderr: go: downloading github.com/rs/xid v1.5.0
test imports
	github.com/rs/xid: module lookup disabled by GOPROXY=off
@gopherbot gopherbot added Tools gopls labels Apr 27, 2022
@gopherbot gopherbot added this to the Unreleased milestone Apr 27, 2022
@bcmills
Copy link
Member

@bcmills bcmills commented Apr 28, 2022

In general, when you combine two modules in a workspace the selected versions of the modules that provide the packages they import may be higher than the versions selected in the individual modules.

(One module in the workspace can upgrade an otherwise-irrelevant dependency vendored at a lower version in a another module, in which case the package at the correct version won't be found in any of the vendor directories at all.)

@bcmills
Copy link
Member

@bcmills bcmills commented Apr 28, 2022

This is expected in the current design of workspaces and not feasible to change.

Unfortunately, workspaces that depend on private repos need to be able to fetch those repos into the module cache.

@bcmills bcmills closed this as completed Apr 28, 2022
@hyangah
Copy link
Contributor

@hyangah hyangah commented Apr 28, 2022

Agree that combining vendor info from multiple modules from a workspace is infeasible.

Interestingly, however, when there is only one module in a workspace, I see go list seems to work (I think) but gopls is unhappy.

Here is a case that is confusing to me.

--- go.work ---
go 1.18

use ./proj

--- proj/go.mod --
module proj

go 1.18

require golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3

require golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect

-- proj/main.go --
package main

import _ "golang.org/x/mod/modfile"

func main() {}

With the above content, prepare the following structure and run GOWORK=off go mod vendor from proj directory.

go.work
proj/
   go.mod
   go.sum
   main.go
   vendor/
      ...

Then,

$ export GOMODCACHE=`$(mktemp -d)`
$ export GOPROXY=off
$ go list -f "{{.Dir}}" golang.org/x/mod/modfile
/tmp/repro/proj/vendor/golang.org/x/mod/modfile
$ cd proj; go list -f "{{.Dir}}" golang.org/x/mod/modfile
/tmp/repro/proj/vendor/golang.org/x/mod/modfile

On the other hand, gopls fails to locate the dependency.

could not import golang.org/x/mod/modfile (no required module provides package "golang.org/x/mod/modfile") compiler [BrokenImport]

I cannot explain the difference in go and gopls's behavior.
@findleyr

@hyangah hyangah reopened this Apr 29, 2022
@hyangah hyangah added the NeedsInvestigation label Apr 29, 2022
@findleyr
Copy link
Contributor

@findleyr findleyr commented Apr 29, 2022

Reopening to investigate if we can mitigate this (partially) from gopls.

@suzmue suzmue removed this from the Unreleased milestone May 2, 2022
@suzmue suzmue added this to the gopls/on-deck milestone May 2, 2022
@findleyr findleyr added the gopls/workspace label May 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
gopls/workspace gopls modules NeedsInvestigation Tools Unfortunate
Projects
None yet
Development

No branches or pull requests

6 participants