Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: error isn't properly reported when running go list on cgo package with missing pkg-config #35882

Open
muirdm opened this issue Nov 27, 2019 · 16 comments
Milestone

Comments

@muirdm
Copy link

@muirdm muirdm commented Nov 27, 2019

What version of Go are you using (go version)?

$ go version
go version go1.13.3 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/muir/Library/Caches/go-build"
GOENV="/Users/muir/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/muir/projects//go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/_q/1xpb0k992ll803xsldpx0qyr0000gn/T/go-build252385107=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I opened a non-cgo package in our project. The package does not transitively depend on any cgo packages.

What did you expect to see?

I expected gopls to load the package with no issue.

What did you see instead?

gopls failed to create the view due to a go list "missing pkg-config" error.

LSP :: Error loading workspace folders (expected 1, got 0)
failed to load view for file:///Users/muir/projects/<project>/go/src: error loading packages: go [list -e -json -compiled=true -test=true -export=false -deps=true -find=false -- ./...]: exit status 2: # pkg-config --cflags  -- <bunch of libraries>
pkg-config: exec: "pkg-config": executable file not found in $PATH

Elsewhere, our project has a cgo package that uses the cgo pkg-config directive (e.g. // #cgo pkg-config: some-library). My laptop doesn't currently have pkg-config installed.

I assume the error comes from the go list ./... we do now when creating a view.

@gopherbot gopherbot added this to the Unreleased milestone Nov 27, 2019
@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Nov 27, 2019

Thank you for filing a gopls issue! Please take a look at the Troubleshooting guide, and make sure that you have provided all of the relevant information here.

@stamblerre

This comment has been minimized.

Copy link
Contributor

@stamblerre stamblerre commented Nov 27, 2019

Thanks for reporting. This should probably be handled more gracefully in go/packages. Assigning to @matloob.

@stamblerre stamblerre modified the milestones: Unreleased, gopls v1.0 Dec 4, 2019
@matloob

This comment has been minimized.

Copy link
Contributor

@matloob matloob commented Dec 6, 2019

I can reproduce the underlying go list issue with the following layout:

foo/go.mod:

module foo

go1.13

foo/foo.go:

package foo

// #cgo pkg-config: bar
import "C"

output of go list -json -compiled . in directory foo:

{
	"Dir": "/Users/matloob/Desktop/foo",
	"ImportPath": "foo",
	"Name": "foo",
	"Root": "/Users/matloob/Desktop/foo",
	"Module": {
		"Path": "foo",
		"Main": true,
		"Dir": "/Users/matloob/Desktop/foo",
		"GoMod": "/Users/matloob/Desktop/foo/go.mod",
		"GoVersion": "1.13"
	},
	"Match": [
		"."
	],
	"Stale": true,
	"StaleReason": "build ID mismatch",
	"CgoFiles": [
		"foo.go"
	],
	"CgoPkgConfig": [
		"bar"
	],
	"Imports": [
		"C",
		"unsafe",
		"runtime/cgo",
		"syscall"
	],
	"Deps": [
		"errors",
		"internal/bytealg",
		"internal/cpu",
		"internal/oserror",
		"internal/race",
		"internal/reflectlite",
		"runtime",
		"runtime/cgo",
		"runtime/internal/atomic",
		"runtime/internal/math",
		"runtime/internal/sys",
		"sync",
		"sync/atomic",
		"syscall",
		"unsafe"
	]
}

go list exits with an exit status of 2.

I'm going to send a cl to suppress the error in go/packages. There are still two problems in go list. The first is that -e isn't being respected. The second is that no error is being generated in the package output by go list.

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Dec 6, 2019

Change https://golang.org/cl/210202 mentions this issue: go/packages: suppress go list -e error when pkg-config can't be run

@matloob

This comment has been minimized.

Copy link
Contributor

@matloob matloob commented Dec 6, 2019

I'm going to repurpose this bug to fix the issue in go list.

Note that even when the issue is fixed in go list, if go/packages or gopls are operating on a project that has a dependency on a cgo package that cgo can't successfully be run on, they won't have access to the full set of sources (which need to be generated by cgo). So they might still have problems for those projects.

Of course if none of the packages you're working on have a dependency on the offending package, things should work.

@matloob matloob changed the title x/tools/gopls: gopls can't start without pkg-config cmd/go: error isn't properly reported when running go list on cgo package with missing pkg-config Dec 6, 2019
@matloob matloob added the GoCommand label Dec 6, 2019
@muirdm

This comment has been minimized.

Copy link
Author

@muirdm muirdm commented Dec 6, 2019

Also note that even after installing pkg-config, go/packages still failed in a similar way because the C library wasn't installed.

@matloob

This comment has been minimized.

Copy link
Contributor

@matloob matloob commented Dec 6, 2019

What was the error it failed with? we're probably going to have to put another special case in go/packages to ignore that error...

@muirdm

This comment has been minimized.

Copy link
Author

@muirdm muirdm commented Dec 6, 2019

2019/12/06 11:16:40 failed to load snapshot: go [list -e -json -compiled=true -test=true -export=false -deps=true -find=false -- ./...]: exit status 2: # pkg-config --cflags  -- libavdevice libavformat libavfilter libavcodec libswscale libavutil libavdevice libavformat libavfilter libavcodec libswscale libavutil libavdevice libavformat libavfilter libavcodec libswscale libavutil libavdevice libavformat libavfilter libavcodec libswscale libavutil
Package libavdevice was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavdevice.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavdevice' found
Package libavformat was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavformat.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavformat' found
Package libavfilter was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavfilter.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavfilter' found
Package libavcodec was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavcodec.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavcodec' found
Package libswscale was not found in the pkg-config search path.
Perhaps you should add the directory containing `libswscale.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libswscale' found
Package libavutil was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavutil.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavutil' found
Package libavdevice was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavdevice.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavdevice' found
Package libavformat was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavformat.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavformat' found
Package libavfilter was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavfilter.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavfilter' found
Package libavcodec was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavcodec.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavcodec' found
Package libswscale was not found in the pkg-config search path.
Perhaps you should add the directory containing `libswscale.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libswscale' found
Package libavutil was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavutil.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavutil' found
Package libavdevice was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavdevice.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavdevice' found
Package libavformat was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavformat.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavformat' found
Package libavfilter was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavfilter.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavfilter' found
Package libavcodec was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavcodec.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavcodec' found
Package libswscale was not found in the pkg-config search path.
Perhaps you should add the directory containing `libswscale.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libswscale' found
Package libavutil was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavutil.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavutil' found
Package libavdevice was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavdevice.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavdevice' found
Package libavformat was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavformat.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavformat' found
Package libavfilter was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavfilter.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavfilter' found
Package libavcodec was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavcodec.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavcodec' found
Package libswscale was not found in the pkg-config search path.
Perhaps you should add the directory containing `libswscale.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libswscale' found
Package libavutil was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavutil.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavutil' found
pkg-config: exit status 1
# pkg-config --cflags  -- libavdevice libavformat libavfilter libavcodec libswscale libavutil libavdevice libavformat libavfilter libavcodec libswscale libavutil libavdevice libavformat libavfilter libavcodec libswscale libavutil libavdevice libavformat libavfilter libavcodec libswscale libavutil
Package libavdevice was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavdevice.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavdevice' found
Package libavformat was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavformat.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavformat' found
Package libavfilter was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavfilter.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavfilter' found
Package libavcodec was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavcodec.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavcodec' found
Package libswscale was not found in the pkg-config search path.
Perhaps you should add the directory containing `libswscale.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libswscale' found
Package libavutil was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavutil.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavutil' found
Package libavdevice was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavdevice.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavdevice' found
Package libavformat was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavformat.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavformat' found
Package libavfilter was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavfilter.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavfilter' found
Package libavcodec was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavcodec.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavcodec' found
Package libswscale was not found in the pkg-config search path.
Perhaps you should add the directory containing `libswscale.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libswscale' found
Package libavutil was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavutil.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavutil' found
Package libavdevice was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavdevice.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavdevice' found
Package libavformat was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavformat.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavformat' found
Package libavfilter was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavfilter.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavfilter' found
Package libavcodec was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavcodec.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavcodec' found
Package libswscale was not found in the pkg-config search path.
Perhaps you should add the directory containing `libswscale.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libswscale' found
Package libavutil was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavutil.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavutil' found
Package libavdevice was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavdevice.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavdevice' found
Package libavformat was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavformat.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavformat' found
Package libavfilter was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavfilter.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavfilter' found
Package libavcodec was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavcodec.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavcodec' found
Package libswscale was not found in the pkg-config search path.
Perhaps you should add the directory containing `libswscale.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libswscale' found
Package libavutil was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavutil.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavutil' found
pkg-config: exit status 1

	directory = file:///Users/muir/projects/<project>/go/src
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x16a69d5]

goroutine 136 [running]:
golang.org/x/tools/internal/lsp/cache.(*snapshot).View(0x0, 0xc00018cd80, 0xc0000ce300)
	/Users/muir/projects/tools/internal/lsp/cache/snapshot.go:61 +0x5
golang.org/x/tools/internal/lsp.(*Server).diagnoseSnapshot(0xc000278f60, 0x19cc4c0, 0x0)
	/Users/muir/projects/tools/internal/lsp/diagnostics.go:20 +0x69
created by golang.org/x/tools/internal/lsp.(*Server).addFolders
	/Users/muir/projects/tools/internal/lsp/general.go:173 +0x600
@matloob

This comment has been minimized.

Copy link
Contributor

@matloob matloob commented Dec 6, 2019

I'm abandoning the change. We've decided that the cost of the change is greater than the benefit. Because it's impossible to determine which package the error belongs to, we'd have to drop the error (or add errors to all packages) to work around it.

In general it's hard for go/packages to work correctly for code that doesn't build.

@muirdm

This comment has been minimized.

Copy link
Author

@muirdm muirdm commented Dec 6, 2019

When "go list" produces the cgo error and exits uncleanly it still also dumps what seems to be the normal JSON results (minus the cgo generated files). What would happen if we ignored the error and continued using the results?

If that is not tenable, is there some way we can at least make gopls able to start in the case of unbuildable cgo package?

@stamblerre

This comment has been minimized.

Copy link
Contributor

@stamblerre stamblerre commented Dec 6, 2019

Did CL 209420 help with this at all, or does gopls still fail in this case? If it does, I can add some workarounds in gopls to handle it.

@muirdm

This comment has been minimized.

Copy link
Author

@muirdm muirdm commented Dec 6, 2019

I uninstalled ffmpeg and still got the above panic (on master). Should be easy to reproduce by adding something like this to a file and then restarting gopls:

// #cgo pkg-config: some-library
import "C"
@stamblerre

This comment has been minimized.

Copy link
Contributor

@stamblerre stamblerre commented Dec 6, 2019

Ah, I just fixed that panic in CL 210215.

@muirdm

This comment has been minimized.

Copy link
Author

@muirdm muirdm commented Dec 6, 2019

Hot dog! Yes, seems to start up fine now.

@stamblerre

This comment has been minimized.

Copy link
Contributor

@stamblerre stamblerre commented Dec 6, 2019

Perfect! So it sounds like we will wait for this fix to be implemented in go list.

@jayconrod jayconrod added the NeedsFix label Dec 9, 2019
@jayconrod jayconrod removed this from the Unreleased milestone Dec 9, 2019
@jayconrod jayconrod added this to the Go1.15 milestone Dec 9, 2019
@jayconrod

This comment has been minimized.

Copy link
Contributor

@jayconrod jayconrod commented Dec 9, 2019

I think this is one instance of a broader problem: execution errors (from running the compiler, linker, or other tools) are not connected back to go list -e output.

Fixing that would be too complicated for 1.14 at this point in the freeze. We should consider it for 1.15 though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.