Scan kubernetes/kubernetes with govulncheck

[PushkarJ]

Background: Today we have scanning implemented using snyk. It has worked quite well with addition of some smart optimization to reduce false positives.

Go team recently released https://go.dev/blog/govulncheck v1.0.0. It promises to provide prioritized vulnerability scanning for CVEs that affect the functions that the code is calling. This is promising in terms of having a really really low false positive since most vulnerability scan reports are in general notoriously hard to wrangle.

Usecases: We have three real workflows for injecting this type of scanning:

1. On k/k PRs: Create a diff between vulnerability scan report run on master branch and the one run on HEAD (current) branch. If the diff is non-zero, fail the pre-merge test. This can be run on symbol and module level depending on context of the PR
2. On k/k master periodically: Run every few hours to get a sense of vulnerability impact for tip of the contributions
3. On k/k release branches: Run every few hours to get a sense of vulnerability impact for release branches so cherry-picks can be created as needed

Tasklist

• [govulncheck] Pre-submit Prow Job for govulncheck #99
• [govulncheck] Periodic Prow Job for govulncheck #100

How it works

Example output on August 4 2023

demo$ govulncheck ./...
Using go1.20.6 and govulncheck@v1.0.0 with vulnerability data from https://vuln.go.dev (last modified 2023-08-02 20:33:39 +0000 UTC).

Scanning your code and 1968 packages across 204 dependent modules for known vulnerabilities...


Vulnerability #1: GO-2023-1987
    Large RSA keys can cause high CPU usage in crypto/tls
  More info: https://pkg.go.dev/vuln/GO-2023-1987
  Standard library
    Found in: crypto/tls@go1.20.6
    Fixed in: crypto/tls@go1.21rc4
    Example traces found:
      #1: pkg/kubelet/server/server.go:234:24: server.ListenAndServePodResources calls grpc.Server.Serve, which eventually calls tls.Conn.Handshake
      #2: pkg/proxy/healthcheck/proxier_health.go:179:24: healthcheck.proxierHealthServer.Run calls http.Server.Serve, which eventually calls tls.Conn.HandshakeContext
      #3: test/e2e/framework/network/utils.go:1026:25: network.PokeHTTP calls io.ReadAll, which calls tls.Conn.Read
      #4: cmd/kubeadm/app/preflight/checks.go:544:13: preflight.SystemVerificationCheck.Check calls bufio.Writer.Flush, which calls tls.Conn.Write
      #5: test/utils/apiserver/testapiserver.go:73:59: apiserver.writeKubeConfigForWardleServerToKASConnection calls cert.GetServingCertificatesForURL, which eventually calls tls.Dial
      #6: test/e2e/framework/websocket/websocket_util.go:61:29: websocket.OpenWebSocketForURL calls websocket.DialConfig, which eventually calls tls.DialWithDialer
      #7: test/images/agnhost/inclusterclient/main.go:100:24: inclusterclient.debugRt.RoundTrip calls spdy.SpdyRoundTripper.RoundTrip, which eventually calls tls.Dialer.DialContext

=== Informational ===

Found 1 vulnerability in packages that you import, but there are no call
stacks leading to the use of this vulnerability. You may not need to
take any action. See https://pkg.go.dev/golang.org/x/vuln/cmd/govulncheck
for details.

Vulnerability #1: GO-2023-1988
    Improper rendering of text nodes in golang.org/x/net/html
  More info: https://pkg.go.dev/vuln/GO-2023-1988
  Module: golang.org/x/net
    Found in: golang.org/x/net@v0.12.0
    Fixed in: golang.org/x/net@v0.13.0

Your code is affected by 1 vulnerability from the Go standard library.

Some more examples from @liggitt https://gist.github.com/liggitt/4674c7eb194738989183abf08feb333f

Open Questions:

These questions need to be discussed and reached a consensus on amongst K8s SRC, SIG Architecture, Release and Security

1. Do we make the scan results available in the CI output of the tests assuming the publicly available tool can not provide any new info than what is available in the CI script ?
2. Do we create two separates tests one for triaged but not yet merged fixes & one for triaged and merged fixes (Triaged issues refer to CVE IDs which have been mentioned in GitHub Issues in k/k)
3. How often does the vulnerability database gets updated?
4. Does it have any differences in refresh cycles for vulnerabilities in go standard libraries (e.g. crypto/tls) v/s go packages in https://pkg.go.dev/ (e.g. golang.org/x/net v0.12.0)
5. Is there a definitive GOOS and GOARCH support we need to adhere to for scanning as the scanning can give different results based on the values for these env vars?

Post-script
In case there is anyone worried about the above output:

GO-2023-1987 fixed in:

• Build Go 1.20.7 images release#3189
• Image promotion for build-image v1.27.0-go1.20.7-bullseye.0 / v1.25.0-go1.20.7-bullseye.0 / v1.26.0-go1.20.7-bullseye.0 / v1.24.0-go1.20.7-bullseye.0 k8s.io#5679
• Image promotion for build-image v2.3.1-go1.20.7-bullseye.0 k8s.io#5680
• Image promotion for releng 1.20.7-bookworm-0 k8s.io#5681

GO-2023-1988 fixed in:

• [CVE-2023-3978] .*: bump golang.org/x/net to v0.13.0 kubernetes#119725

Previous discussions:

• v1.0.0 blog post,
• slack,
• sig-security discussion on 2023-07-27

/sig security architecture release
/committee security-response

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[PushkarJ]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[PushkarJ]

/remove-lifecycle rotten