Checksum in sum.golang.org is different from download without proxy/checksum database (GOPRIVATE=*)

[piotrkubisa]

References

The GOPRIVATE environment variable controls which modules the go command considers to be private (not available publicly) and should therefore not use the proxy or checksum database.
-- https://pkg.go.dev/cmd/go

What?

Whether GOPRIVATE=* environment has been set, the checksum in go.sum for berglas dependency is different than if I would not set GOPRIVATE (standard Go installation). This issue makes unable to have defined custom GOPRIVATE across the team and/or CI environments, because Go compiler will deny any further operations.

While, I rely on numerous go-modules (including various Gcloud SDKs) I have only encountered this problem in berglas. The fact that the berglas manages secrets, this issue might be a security-related problem, because default sum.golang.org offers different version of berglas than actually is distributed over github.com.

How to reproduce?

In a secret gist I have prepared a sample project that imports github.com/GoogleCloudPlatform/berglas and prints out the go.sum, depending whether GOPRIVATE=* was or wasn't defined.

In the diff(1) output presented below, I compare the computed go.sum file with GOPRIVATE=* set and without respectively.

diff out/with-goprivate.txt out/no-goprivate.txt 

57c57
< github.com/GoogleCloudPlatform/berglas v0.6.2 h1:GeTSHX7B/eIONuwnVuh92I3f0yAmdSz6RdAYP5/efQI=
---
> github.com/GoogleCloudPlatform/berglas v0.6.2 h1:mgcdWF53ltZA0hBGWrpBAjuKh5VnQPxHRlFRoHd5tpA=

If you would like to test it on your machine, please take a look in my gist - https://gist.github.com/piotrkubisa/4dbba8d36bf3748944489dc523eefead. I hope example presenting sample code will be more clear for you.

main.go

package main

import (
	"fmt"
	"os"

	"github.com/GoogleCloudPlatform/berglas/pkg/berglas"
	"github.com/sirupsen/logrus"
)

var _ *berglas.ListResponse = nil

func main() {
	fmt.Println("Hello World!")

	logrus.Println("Testing")

	contents, err := os.ReadFile("go.sum")
	if err != nil {
		panic(err)
	}
	fmt.Println(string(contents))
}

Dockerfile

FROM golang:1.17 AS builder

WORKDIR /src
COPY main.go .

# comment/uncomment line below to see a difference in go.sum
# ENV GOPRIVATE=* 

RUN go mod init my-app
RUN go mod tidy --compat=1.17
RUN cat go.mod
RUN cat go.sum

RUN CGO_ENABLED=0 go build -o /bin/app

FROM gcr.io/distroless/static:latest
COPY --from=builder /bin/app /bin/app
COPY --from=builder /src/go.sum /go.sum
ENTRYPOINT ["/bin/app"]

[sethvargo]

I have no idea where to even begin to debug something like that.

[sethvargo]

I tried running the script in your gist on 3 different networks (home, cellular, coffeeshop) to try and reproduce.

First, I tried with no other dependencies besides berglas (removed "github.com/sirupsen/logrus"). On all three networks, I go no diff.

Next, I tried adding "github.com/sirupsen/logrus" back in, but got the same result.

My suspicion is that your proxy, network, ISP, or something higher up the chain is modifying the response.

[piotrkubisa]

Logrus has nothing to do with the issue. It presents a completely other dependency that does not seem to have different checksum, whether the GOPRIVATE=* is set or not. It also takes part in verification if my network setup has some proxy or doesn't. Because, the checksum is exact for both scenarios, I think I don't have any unknown proxy of my ISP.

Also, because we start blaming someone's other network, which likely has nothing to do with my problem, I have prepared you two GitHub workflows, one with GOPRIVATE=* and other without. I find GitHub-managed workflow runners as a neutral ground and good place for more investigation. Here are results, which confirms my thesis - I am not only one, who can be haunted by this problem:

Without GOPRIVATE

• Workflow definition: https://github.com/piotrkubisa/berglas-checksum/blob/master/.github/workflows/no-goprivate.yml#L6-L9
• Test run: https://github.com/piotrkubisa/berglas-checksum/runs/4475900810?check_suite_focus=true#step:6:65

github.com/GoogleCloudPlatform/berglas v0.6.2 h1:mgcdWF53ltZA0hBGWrpBAjuKh5VnQPxHRlFRoHd5tpA=
github.com/GoogleCloudPlatform/berglas v0.6.2/go.mod h1:LEKpytS5wf+P8OGenFlsVmi/uwqcjkFQTH8wZABaCgI=

With GOPRIVATE=*

• Workflow definition: https://github.com/piotrkubisa/berglas-checksum/blob/master/.github/workflows/with-goprivate.yml#L6-L10
• Test run: https://github.com/piotrkubisa/berglas-checksum/runs/4475900876?check_suite_focus=true#step:6:66

github.com/GoogleCloudPlatform/berglas v0.6.2 h1:GeTSHX7B/eIONuwnVuh92I3f0yAmdSz6RdAYP5/efQI=
github.com/GoogleCloudPlatform/berglas v0.6.2/go.mod h1:LEKpytS5wf+P8OGenFlsVmi/uwqcjkFQTH8wZABaCgI=

[sethvargo]

Alright, I reworked your example a bit to eliminate as many variables as possible and was still able to reproduce it (workflow).

I downloaded the cached binary from https://proxy.golang.org/github.com/%21google%21cloud%21platform/berglas/@v/v0.6.2.zip, unzipped it, and compared it to the v0.6.2 tag. It looks like none of the examples/ are included:

HEAD detached at v0.6.2
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	deleted:    examples/appengine/go/.gcloudignore
	deleted:    examples/appengine/go/.gitignore
	deleted:    examples/appengine/go/README.md
	deleted:    examples/appengine/go/app.yaml
	deleted:    examples/appengine/go/go.mod
	deleted:    examples/appengine/go/go.sum
	deleted:    examples/appengine/go/main.go
	deleted:    examples/appengineflex/go/.gitignore
	deleted:    examples/appengineflex/go/Dockerfile
	deleted:    examples/appengineflex/go/README.md
	deleted:    examples/appengineflex/go/app.yaml
	deleted:    examples/appengineflex/go/go.mod
	deleted:    examples/appengineflex/go/go.sum
	deleted:    examples/appengineflex/go/main.go
	deleted:    examples/cloudfunctions/go/README.md
	deleted:    examples/cloudfunctions/go/fn.go
	deleted:    examples/cloudfunctions/go/go.mod
	deleted:    examples/cloudfunctions/go/go.sum
	deleted:    examples/cloudrun/go/Dockerfile
	deleted:    examples/cloudrun/go/README.md
	deleted:    examples/cloudrun/go/go.mod
	deleted:    examples/cloudrun/go/go.sum
	deleted:    examples/cloudrun/go/main.go
	deleted:    examples/kubernetes/README.md
	deleted:    examples/kubernetes/deploy/sample.yaml
	deleted:    examples/kubernetes/deploy/webhook.yaml
	deleted:    examples/kubernetes/go.mod
	deleted:    examples/kubernetes/go.sum
	deleted:    examples/kubernetes/main.go

This makes me think that the proxy doesn't include that directory, either because it declares its own go.sum/mod files, or because it is named examples (which seems weird).

Nonetheless, I don't see anything malicious here. If you believe the Go module proxy is modifying packages in ways that it shouldn't, I would recommend using the contact information at the bottom of https://proxy.golang.org/.

[piotrkubisa]

Okay, that sounds great!

I have strong suspicion towards examples/ directory, which you have named as possible root of the problem. I have done two little changes to the berglas repository in my fork:

• v0.6.3 - rm -rf examples/ completely. The go.sum file started to look exactly the same in both scenarios.
• v0.6.4 - I have reverted changes from v0.6.3 and just renamed the examples dir (mv examples renamed-examples-dir). The workflow says, the checksum is identical in both scenarios too.

$ export GOPRIVATE=*
$ go mod download -json -x github.com/piotrkubisa/berglas@v0.6.4
{
        "Path": "github.com/piotrkubisa/berglas",
        "Version": "v0.6.4",
        "Info": "/home/piotr/go/pkg/mod/cache/download/github.com/piotrkubisa/berglas/@v/v0.6.4.info",
        "GoMod": "/home/piotr/go/pkg/mod/cache/download/github.com/piotrkubisa/berglas/@v/v0.6.4.mod",
        "Zip": "/home/piotr/go/pkg/mod/cache/download/github.com/piotrkubisa/berglas/@v/v0.6.4.zip",
        "Dir": "/home/piotr/go/pkg/mod/github.com/piotrkubisa/berglas@v0.6.4",
        "Sum": "h1:d6xGda/YPfMBD3q6Qbpgqi5slRlQ4ULJ5w/8nPW4fK4=",
        "GoModSum": "h1:LEKpytS5wf+P8OGenFlsVmi/uwqcjkFQTH8wZABaCgI="
}
$ unset GOPRIVATE
$ go mod download -json -x github.com/piotrkubisa/berglas@v0.6.4
{
        "Path": "github.com/piotrkubisa/berglas",
        "Version": "v0.6.4",
        "Info": "/home/piotr/go/pkg/mod/cache/download/github.com/piotrkubisa/berglas/@v/v0.6.4.info",
        "GoMod": "/home/piotr/go/pkg/mod/cache/download/github.com/piotrkubisa/berglas/@v/v0.6.4.mod",
        "Zip": "/home/piotr/go/pkg/mod/cache/download/github.com/piotrkubisa/berglas/@v/v0.6.4.zip",
        "Dir": "/home/piotr/go/pkg/mod/github.com/piotrkubisa/berglas@v0.6.4",
        "Sum": "h1:d6xGda/YPfMBD3q6Qbpgqi5slRlQ4ULJ5w/8nPW4fK4=",
        "GoModSum": "h1:LEKpytS5wf+P8OGenFlsVmi/uwqcjkFQTH8wZABaCgI="
}

Would the rename from examples/ to demo/ will fix this problem?

[github-actions]

This issue is stale because it has been open for 14 days with no
activity. It will automatically close after 7 more days of inactivity.

[sethvargo]

Would the rename from examples/ to demo/ will fix this problem?

It's possible, but I "examples" is a pretty common name. I really think a bug should be opened against the module registry for this.