Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/vuln: use govulncheck in bazel #61494

Open
ukai opened this issue Jul 21, 2023 · 9 comments
Open

x/vuln: use govulncheck in bazel #61494

ukai opened this issue Jul 21, 2023 · 9 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. vulncheck or vulndb Issues for the x/vuln or x/vulndb repo

Comments

@ukai
Copy link
Contributor

ukai commented Jul 21, 2023

What version of Go are you using (go version)?

$ go version
go version go1.21-20230628-RC02 cl/544161750 +8b5fe5980c X:fieldtrack,boringcrypto linux/amd64

Does this issue reproduce at the latest version of golang.org/x/vuln?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/usr/local/google/home/ukai/.cache/go-build'
GOENV='/usr/local/google/home/ukai/.config/go/env'
GOEXE=''
GOEXPERIMENT='fieldtrack,boringcrypto'
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/usr/local/google/home/ukai/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/usr/local/google/home/ukai/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/lib/google-golang'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/lib/google-golang/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21-20230628-RC02 cl/544161750 +8b5fe5980c X:fieldtrack,boringcrypto'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3493584143=/tmp/go-build -gno-record-gcc-switches'

What did you do?

install govulncheck and run it in bazel workspace.

e.g.

$  git clone https://github.com/bazelbuild/reclient
Cloning into 'reclient'...
remote: Enumerating objects: 640, done.
remote: Counting objects: 100% (640/640), done.
remote: Compressing objects: 100% (410/410), done.
remote: Total 640 (delta 211), reused 628 (delta 199), pack-reused 0
Receiving objects: 100% (640/640), 757.42 KiB | 12.84 MiB/s, done.
Resolving deltas: 100% (211/211), done.
$ cd reclient 
$ govulncheck ./...

What did you expect to see?

check go vulnerabilities in workspace

What did you see instead?

$ govulncheck ./..
Using go1.21-20230628-RC02 cl/544161750 +8b5fe5980c X:fieldtrack,boringcrypto and govulncheck@v1.0.0 with vulnerability data from https://vuln.go.dev (last modified 2023-07-13 22:19:53 +0000 UTC).

govulncheck: no go.mod file

govulncheck only works with Go modules. Try navigating to your module directory.
Otherwise, run go mod init to make your project a module.

See https://go.dev/doc/modules/managing-dependencies for more information.

how can we use govulncheck for bazel go code?

@ukai ukai added the vulncheck or vulndb Issues for the x/vuln or x/vulndb repo label Jul 21, 2023
@gopherbot gopherbot modified the milestones: Unreleased, vuln/unplanned Jul 21, 2023
@seankhliao

This comment was marked as resolved.

@seankhliao seankhliao closed this as not planned Won't fix, can't repro, duplicate, stale Jul 21, 2023
@ukai
Copy link
Contributor Author

ukai commented Jul 21, 2023

I think it's not question, but feature request?

@timothy-king
Copy link
Contributor

This is a feature request.

Bazel is not currently supported for govulncheck. A big barrier right now is how to translate the bazel libraries to module versions for a bazel build. govulncheck cannot give meaningful output unless given such a mapping so that it to compare against its database.

how can we use govulncheck for bazel go code?

Given the above, the direct answer to your question today is rewrite the package so that it can be built by go build and has an accurate go.mod file. (I am aware this is likely not very helpful.)

(I would recommend reworking the issue a bit so it is obviously in the form of an feature request instead of a question.)

@timothy-king timothy-king reopened this Jul 26, 2023
@mknyszek mknyszek added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 7, 2023
@mknyszek
Copy link
Contributor

mknyszek commented Aug 7, 2023

CC @golang/vulndb

@loeffel-io

This comment was marked as spam.

@timothy-king
Copy link
Contributor

@loeffel-io FYI you can use emoji voting 👍 instead of +1.

@sluongng
Copy link

Technically, most of the recent go tools, including govulncheck leverage the package driver protocol documented in https://pkg.go.dev/golang.org/x/tools/go/packages#hdr-The_driver_protocol

This allows build tools, such as Bazel or Buck2, to provide package information to the go tools. For Bazel rules_go, a basic package driver is provided via https://github.com/bazelbuild/rules_go/wiki/Editor-and-tool-integration

I think the current problem with this is that either govulncheck or the basic driver in rules_go provide any diagnostic outputs. Even with -show verbose set, all I got out from a govulncheck run is

govulncheck: loading packages: root package @//some/pkg:pkg is missing

or

govulncheck: loading packages:
There are errors with the provided package patterns:

/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/plugin/plugin.go:80:9: undefined: open
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/plugin/plugin.go:88:9: undefined: lookup
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/testing/cover.go:50:18: undefined: goexperiment.CoverageRedesign
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/testing/cover.go:85:18: undefined: goexperiment.CoverageRedesign
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/testing/newcover.go:36:19: undefined: goexperiment.CoverageRedesign
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/testing/testing.go:669:18: undefined: goexperiment.CoverageRedesign
-: sources missing for package @io_bazel_rules_go//stdlib:internal/coverage/test
-: sources missing for package @io_bazel_rules_go//stdlib:runtime/internal/wasitest
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/builtin/builtin.go:215:6: generic function is missing function body
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/builtin/builtin.go:221:6: generic function is missing function body
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/builtin/builtin.go:251:6: generic function is missing function body
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/builtin/builtin.go:16:6: invalid recursive type: bool refers to itself
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/builtin/builtin.go:26:6: invalid recursive type: uint8 refers to itself
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/builtin/builtin.go:30:6: invalid recursive type: uint16 refers to itself
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/builtin/builtin.go:34:6: invalid recursive type: uint32 refers to itself
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/builtin/builtin.go:38:6: invalid recursive type: uint64 refers to itself
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/builtin/builtin.go:42:6: invalid recursive type: int8 refers to itself
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/builtin/builtin.go:46:6: invalid recursive type: int16 refers to itself
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/builtin/builtin.go:50:6: invalid recursive type: int32 refers to itself
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/builtin/builtin.go:54:6: invalid recursive type: int64 refers to itself
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/builtin/builtin.go:57:6: invalid recursive type: float32 refers to itself
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/builtin/builtin.go:60:6: invalid recursive type: float64 refers to itself
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/builtin/builtin.go:64:6: invalid recursive type: complex64 refers to itself
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/builtin/builtin.go:68:6: invalid recursive type: complex128 refers to itself
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/builtin/builtin.go:73:6: invalid recursive type: string refers to itself
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/builtin/builtin.go:77:6: invalid recursive type: int refers to itself
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/builtin/builtin.go:81:6: invalid recursive type: uint refers to itself
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/builtin/builtin.go:85:6: invalid recursive type: uintptr refers to itself
/private/var/tmp/_bazel_sluongng/06e573a93bc2d6a9cad4ad41f00b4310/external/go_sdk_darwin_arm64/src/builtin/builtin.go:104:6: invalid recursive type: comparable refers to itself
-: sources missing for package @io_bazel_rules_go//stdlib:embed/internal/embedtest

For details on package patterns, see https://pkg.go.dev/cmd/go#hdr-Package_lists_and_patterns.

exit 1

So it would be nice if the verbose mode of govulncheck could add some logging into the packages.Visit(..., preFunc, postFunc) call https://go.googlesource.com/vuln/+/3740f5cb12a3f93b18dbe200c4bcb6256f8586e2/internal/vulncheck/packages.go#209. We could also add additional logging to the driver on rules_go side (separate issue) to help correlate.

Once the package loading problem is solved, we could easily instrument govulncheck with a pre-downloaded database from https://vuln.go.dev/vulndb.zip. This could also be improved if there is a checksum for each db version so Bazel could verify the download more easily.

With the items above resolved, we could easily run govulncheck in a small-medium size Bazel setup. For giant monorepos, more works are probably still needed. But we need to crawl before walking.

@ianthehat
Copy link

The trouble is that vulnerability analysis has to operate at a module level, because that is the unit of versioning and versions are strictly necessary for vulnerability matching.
It is possible that if the gopackages driver filled in the Modules part of the response it could be made to work, but without that it is a non starter.

@sluongng
Copy link

@ianthehat I think that's possible. The current driver in rules_go predates the recent advancements under x/tools. So the raw JSON format of the driver protocol is compatible but is missing some of the new fields such as Modules.

It would be a nontrivial amount of work to fully support all the NeedModule LoadMode https://pkg.go.dev/golang.org/x/tools/go/packages#LoadMode on the rules_go side though. The version information will probably need to trickle down: from the go_repository declaration from Gazelle, down to rules_go, then down to the driver.

cc: @fmeum @tyler-french @linzhp

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. vulncheck or vulndb Issues for the x/vuln or x/vulndb repo
Projects
None yet
Development

No branches or pull requests

8 participants