Athens does not calculate the correct checksum #1881

idefixcert · 2023-09-05T09:34:09Z

Describe the bug
Maybe this is not the right forum, but I am not sure where to ask else.

I use a docker athene container and another fresh golang container.
In the golang container I setup athene container as goproxy.
when I run go install github.com/mgechev/revive@latest I see an error message

ERROR[9:17AM]: github.com/mgechev/revive@v1.3.3: verifying module: checksum mismatch
	downloaded: h1:H5KpQYVmht69pg47kbIg/E4S1jA1yfFBEwy7Zv3uDX8=
	sum.golang.org: h1:GUWzV3g185agbHN4ZdaQvR6zrLVYTUSA2ktvIinivK0=

but if I do not use athene as a proxy I can't see this error.

Error Message
If applicable add error message to help explain your problem or console output enclose in triple back-ticks eg,

```console
INFO[9:17AM]: incoming request	http-method=GET http-path=/github.com/mgechev/@v/list http-status=404 request-id=5a1e4cb1-5b83-4a29-ba1e-bf97198a25c1
INFO[9:17AM]: incoming request	http-method=GET http-path=/github.com/mgechev/revive/@v/list http-status=200 request-id=468b8f6c-4062-4b86-87d1-6422e58b9936
ERROR[9:17AM]: github.com/mgechev/revive@v1.3.3: verifying module: checksum mismatch
	downloaded: h1:H5KpQYVmht69pg47kbIg/E4S1jA1yfFBEwy7Zv3uDX8=
	sum.golang.org: h1:GUWzV3g185agbHN4ZdaQvR6zrLVYTUSA2ktvIinivK0=

SECURITY ERROR
This download does NOT match the one reported by the checksum server.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.
```

To Reproduce
Steps to reproduce the behavior:

docker run --rm -p 3000:3000 gomods/athens:v0.12.1

docker run --rm -it golang:latest
export GOPROXY="http://172.17.0.1:3000/";
go install github.com/mgechev/revive@latest

Expected behavior
Same behavior with and without proxy.

Environment (please complete the following information):

OS: ubuntu with the docker containers
Go version : 1.21
Proxy version : v0.12.1

The text was updated successfully, but these errors were encountered:

bozaro · 2023-09-27T09:47:46Z

Same issue with github.com/homeport/dyff@v1.5.7 package (without proxy module downloaded succesfully):

docker run --rm -p 3000:3000 gomods/athens:v0.12.1
GOMODCACHE=$(mktemp -d) GOPROXY=http://127.0.0.1:3000/ go get github.com/homeport/dyff@v1.5.7

INFO[9:50AM]: Exporter not specified. Traces won't be exported	
2023/09/27 09:50:30 Starting application at port :3000
ERROR[9:50AM]: github.com/homeport@v1.5.7: invalid github.com import path "github.com/homeport"	http-method=GET http-path=/github.com/homeport/@v/v1.5.7.info request-id=04a90067-1d2b-4a93-9b74-d12d8e162605 
INFO[9:50AM]: incoming request	http-method=GET http-path=/github.com/homeport/@v/v1.5.7.info http-status=404 request-id=04a90067-1d2b-4a93-9b74-d12d8e162605 
ERROR[9:50AM]: github.com@v1.5.7: invalid github.com import path "github.com"	http-method=GET http-path=/github.com/@v/v1.5.7.info request-id=0078725a-fec4-480e-9c7d-8b4ae2b7cc37 
INFO[9:50AM]: incoming request	http-method=GET http-path=/github.com/@v/v1.5.7.info http-status=404 request-id=0078725a-fec4-480e-9c7d-8b4ae2b7cc37 
ERROR[9:50AM]: github.com/homeport/dyff@v1.5.7: verifying module: checksum mismatch
	downloaded: h1:mwOGR4Zzlr48RoavdeBuDiVpH2PIVKMD19xuvirMDEE=
	sum.golang.org: h1:24IrdSAb2fprR67XHMIagGoc1c4VNBV9ibNFKICsSis=

SECURITY ERROR
This download does NOT match the one reported by the checksum server.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

For more information, see 'go help module-auth'.
	http-method=GET http-path=/github.com/homeport/dyff/@v/v1.5.7.info request-id=b003ccef-4a77-41f4-bc59-a36f9ae1ccc9 
INFO[9:50AM]: incoming request	http-method=GET http-path=/github.com/homeport/dyff/@v/v1.5.7.info http-status=404 request-id=b003ccef-4a77-41f4-bc59-a36f9ae1ccc9

matt0x6F · 2024-04-15T07:28:24Z

The only way I can think of this happening is that a given tag was released twice and Athens has a different one than sum.golang.org knows about. You can try removing the artifact from Athens' disk storage.

bozaro · 2024-04-15T10:32:24Z

You can try removing the artifact from Athens' disk storage.

In this problem reproduction scenario, there is no persistent storage.

This issue is still reproducing.

Start latest athens in one terminal:

docker run --rm -p 3000:3000 gomods/athens:v0.13.3

Try download package in another terminal:

cd $(mktemp -d)
go mod init example.net/foo
GOMODCACHE=$(mktemp -d) GOPROXY=http://127.0.0.1:3000/ go get github.com/homeport/dyff@v1.5.7

Also you can simply try download package with curl:

curl -vf http://127.0.0.1:3000/github.com/homeport/dyff/@v/v1.5.7.mod

You can see error in docker container logs:

INFO[10:30AM]: incoming request	http-method=GET http-path=/github.com/homeport/@v/v1.5.7.info http-status=404 request-id=34806221-e57e-4666-9d02-ef22519517ab 
INFO[10:30AM]: github.com/homeport/dyff@v1.5.7: verifying module: checksum mismatch
	downloaded: h1:mwOGR4Zzlr48RoavdeBuDiVpH2PIVKMD19xuvirMDEE=
	sum.golang.org: h1:24IrdSAb2fprR67XHMIagGoc1c4VNBV9ibNFKICsSis=

SECURITY ERROR
This download does NOT match the one reported by the checksum server.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

For more information, see 'go help module-auth'.

matt0x6F · 2024-04-15T23:48:56Z

Curious! I started up a local version of Athens and was able to reproduce that message with v1.5.7 but it didn't occur with v1.7.1.

Server logs:

athens on  main [$] via 🐳 hermes via 🐹 v1.22.1 
❯ go run ./cmd/proxy/
2024/04/15 16:45:35 Running dev mode with default settings, consult config when you're ready to run in production
INFO[4:45PM]: Exporter not specified. Traces won't be exported
INFO[4:45PM]: Starting application      tcpPort=:3000 
ERROR[4:45PM]: github.com@v1.5.7: invalid github.com import path "github.com"   http-method=GET http-path=/github.com/@v/v1.5.7.info request-id=8131d69d-ca41-4603-9bfa-0abe1b750c75 
INFO[4:45PM]: incoming request  http-method=GET http-path=/github.com/@v/v1.5.7.info http-status=404 request-id=8131d69d-ca41-4603-9bfa-0abe1b750c75 
ERROR[4:45PM]: github.com/homeport@v1.5.7: invalid github.com import path "github.com/homeport" http-method=GET http-path=/github.com/homeport/@v/v1.5.7.info request-id=831bb5bc-9d72-4f52-8871-c50dd503f4ed 
INFO[4:45PM]: incoming request  http-method=GET http-path=/github.com/homeport/@v/v1.5.7.info http-status=404 request-id=831bb5bc-9d72-4f52-8871-c50dd503f4ed 
ERROR[4:45PM]: github.com/homeport/dyff@v1.5.7: verifying module: checksum mismatch
        downloaded: h1:mwOGR4Zzlr48RoavdeBuDiVpH2PIVKMD19xuvirMDEE=
        sum.golang.org: h1:24IrdSAb2fprR67XHMIagGoc1c4VNBV9ibNFKICsSis=

SECURITY ERROR
This download does NOT match the one reported by the checksum server.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

For more information, see 'go help module-auth'.
        http-method=GET http-path=/github.com/homeport/dyff/@v/v1.5.7.info request-id=fe619031-c1cd-4933-be61-4ce9c2da9ea0 
INFO[4:45PM]: incoming request  http-method=GET http-path=/github.com/homeport/dyff/@v/v1.5.7.info http-status=404 request-id=fe619031-c1cd-4933-be61-4ce9c2da9ea0 
ERROR[4:46PM]: github.com@v1.7.1: invalid github.com import path "github.com"   http-method=GET http-path=/github.com/@v/v1.7.1.info request-id=9964a812-80b6-4255-b839-e69166663b39 
INFO[4:46PM]: incoming request  http-method=GET http-path=/github.com/@v/v1.7.1.info http-status=404 request-id=9964a812-80b6-4255-b839-e69166663b39 
ERROR[4:46PM]: github.com/homeport@v1.7.1: invalid github.com import path "github.com/homeport" http-method=GET http-path=/github.com/homeport/@v/v1.7.1.info request-id=6749ffe6-313c-4553-8148-f0708ec1b37f 
INFO[4:46PM]: incoming request  http-method=GET http-path=/github.com/homeport/@v/v1.7.1.info http-status=404 request-id=6749ffe6-313c-4553-8148-f0708ec1b37f 
INFO[4:46PM]: incoming request  http-method=GET http-path=/github.com/homeport/dyff/@v/v1.7.1.info http-status=200 request-id=7f06d6b5-c0bc-44b1-8bf8-9fc39e08610d 
INFO[4:46PM]: incoming request  http-method=GET http-path=/github.com/homeport/dyff/@v/v1.7.1.zip http-status=200 request-id=605e0941-3acd-435d-8475-1781819731d5 
INFO[4:46PM]: incoming request  http-method=GET http-path=/sumdb/sum.golang.org/supported http-status=200 request-id=b8c52051-08c2-4845-a1a5-bbb7d41807c5 
...

Client logs:

athens on  main [$] via 🐳 hermes via 🐹 v1.22.1 
❯ GOPROXY=http://127.0.0.1:3000 go get github.com/homeport/dyff@v1.5.7
go: github.com/homeport/dyff@v1.5.7: reading http://127.0.0.1:3000/github.com/homeport/dyff/@v/v1.5.7.info: 404 Not Found

athens on  main [$] via 🐳 hermes via 🐹 v1.22.1 took 2s 
❯ GOPROXY=http://127.0.0.1:3000 go get github.com/homeport/dyff@v1.7.1
go: downloading github.com/homeport/dyff v1.7.1
go: downloading github.com/BurntSushi/toml v1.3.2
go: downloading github.com/mattn/go-isatty v0.0.19
go: downloading github.com/google/go-cmp v0.6.0
go: downloading github.com/onsi/gomega v1.31.1
go: downloading golang.org/x/sync v0.6.0
go: downloading github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3
go: downloading golang.org/x/tools v0.17.0
go: upgraded github.com/BurntSushi/toml v1.0.0 => v1.3.2
go: upgraded github.com/google/go-cmp v0.5.9 => v0.6.0
go: added github.com/homeport/dyff v1.7.1
go: upgraded github.com/mattn/go-isatty v0.0.14 => v0.0.19
go: upgraded golang.org/x/sync v0.3.0 => v0.6.0

matt0x6F · 2024-04-15T23:56:36Z

The H1 is what's failing, so I'm going to attempt to verify the H1 using this.

matt0x6F · 2024-04-16T02:00:40Z

Well that's pretty curious! I was able to use this tool to reproduce the sum.golang.org H1:

go-checksum . github.com/homeport/dyff@v1.5.7
directory: .
{
        "HashSynthesized": "db822b75201bd9fa6b47aed71cc21a806a1cd5ce1534157d89b3452880ac4a2b",
        "HashSynthesizedBase64": "24IrdSAb2fprR67XHMIagGoc1c4VNBV9ibNFKICsSis=",
        "GoCheckSum": "h1:24IrdSAb2fprR67XHMIagGoc1c4VNBV9ibNFKICsSis="
}

I was not able to reproduce the Checksum Athens produced (h1:mwOGR4Zzlr48RoavdeBuDiVpH2PIVKMD19xuvirMDEE=). All that to say, the bug is verified. I'll see if I can figure out where our checksums may be deviating from what sum.golang.org does.

matt0x6F · 2024-04-16T05:37:13Z

I was able to confirm with go mod download -json that the Sum field is GoChecksum. We download that file in the fetcher by running the Go command.

What's curious is I haven't seen where we get h1:mwOGR4Zzlr48RoavdeBuDiVpH2PIVKMD19xuvirMDEE= yet.

matt0x6F · 2024-04-16T06:19:47Z

The error source is on the op module.downloadModule.

Inside the downloadModule function reporoot will be set to a directory that exists but is empty. The command will run and return the error. I validated that this is using my system Go installation. This is go downloading something that is not what's actually dyff@v1.5.7, or at least not all of it.

This is the error:

"{\n\t\"Path\": \"github.com/homeport/dyff\",\n\t\"Version\": \"v1.5.7\",\n\t\"Error\": \"github.com/homeport/dyff@v1.5.7: verifying module: checksum mismatch\\n\\tdownloaded: h1:mwOGR4Zzlr48RoavdeBuDiVpH2PIVKMD19xuvirMDEE=\\n\\tsum.golang.org: h1:24IrdSAb2fprR67XHMIagGoc1c4VNBV9ibNFKICsSis=\\n\\nSECURITY ERROR\\nThis download does NOT match the one reported by the checksum server.\\nThe bits may have been replaced on the origin server, or an attacker may\\nhave intercepted the download attempt.\\n\\nFor more information, see 'go help ...+435 more"

matt0x6F · 2024-04-16T06:21:00Z

I did find this in the .info of that same directory:

❯ cat pkg/mod/cache/download/github.com/homeport/dyff/@v/v1.5.7.info
{"Version":"v1.5.7","Time":"2023-04-15T20:44:39Z","Origin":{"VCS":"git","URL":"https://github.com/homeport/dyff","Hash":"86654f8e002a69fdabc87ac954f7d0fa11e6b84b","Ref":"refs/tags/v1.5.7"}}

matt0x6F · 2024-04-17T03:30:18Z

Okay, so far more weird. All we're actually doing here is calling go mod download -json github.com/homeport/dyff@v1.5.7. The only environment variables we pass are GOPROXY=direct and GONOSUMDB=. It's using my desktop Go client to do the fetching. It fails with the checksum mismatch on the server and returns a 404 to the client. When I run that same command in my terminal:

❯ GOPROXY=direct go mod download -json github.com/homeport/dyff@v1.5.7
{
        "Path": "github.com/homeport/dyff",
        "Version": "v1.5.7",
        "Error": "github.com/homeport/dyff@v1.5.7: verifying module: checksum mismatch\n\tdownloaded: h1:mwOGR4Zzlr48RoavdeBuDiVpH2PIVKMD19xuvirMDEE=\n\tsum.golang.org: h1:24IrdSAb2fprR67XHMIagGoc1c4VNBV9ibNFKICsSis=\n\nSECURITY ERROR\nThis download does NOT match the one reported by the checksum server.\nThe bits may have been replaced on the origin server, or an attacker may\nhave intercepted the download attempt.\n\nFor more information, see 'go help module-auth'.\n",
        "Info": "/home/matt/go/1.22.1/pkg/mod/cache/download/github.com/homeport/dyff/@v/v1.5.7.info",
        "GoMod": "/home/matt/go/1.22.1/pkg/mod/cache/download/github.com/homeport/dyff/@v/v1.5.7.mod",
        "GoModSum": "h1:AfPyh52slslbDkh1fy0MGIbKTiUy/2xbM1Zos5HZTm4=",
        "Origin": {
                "VCS": "git",
                "URL": "https://github.com/homeport/dyff",
                "Hash": "e2f30e959f2105954276e58a05c1f3526c1d0c0d",
                "Ref": "refs/tags/v1.5.7"
        }

I get the same behavior! Now, when I enable the Go proxy I get a very different response.

❯ GOPROXY=https://proxy.golang.org go mod download -json github.com/homeport/dyff@v1.5.7
{
        "Path": "github.com/homeport/dyff",
        "Version": "v1.5.7",
        "Info": "/home/matt/go/1.22.1/pkg/mod/cache/download/github.com/homeport/dyff/@v/v1.5.7.info",
        "GoMod": "/home/matt/go/1.22.1/pkg/mod/cache/download/github.com/homeport/dyff/@v/v1.5.7.mod",
        "Zip": "/home/matt/go/1.22.1/pkg/mod/cache/download/github.com/homeport/dyff/@v/v1.5.7.zip",
        "Dir": "/home/matt/go/1.22.1/pkg/mod/github.com/homeport/dyff@v1.5.7",
        "Sum": "h1:24IrdSAb2fprR67XHMIagGoc1c4VNBV9ibNFKICsSis=",
        "GoModSum": "h1:AfPyh52slslbDkh1fy0MGIbKTiUy/2xbM1Zos5HZTm4=",
        "Origin": {
                "VCS": "git",
                "URL": "https://github.com/homeport/dyff",
                "Hash": "e2f30e959f2105954276e58a05c1f3526c1d0c0d",
                "Ref": "refs/tags/v1.5.7"
        }
}

The H1's are the same, but the second response returns a Sum field that is a different value from the GoModSum. The Sum field is the actual checksum we're looking for. On top of this, I checked the Go Checksum Database which reads from proxy.golang.org and stores in an append-only format to maintain a Merkle tree of hashes. Unsurprisingly it lines up with what I get when I retrieve the module from the proxy.

So, to recap:

If I pull direct the checksum fails
If I pull from the proxy, everything is kosher

I need to get ahold of the artifact that Go is retrieving with direct and run the checksum on it. That will be my next update.

matt0x6F · 2024-04-17T04:08:12Z

Now we're getting somewhere. I was able to force Go to fetch the module

❯ GOPROXY=direct GOSUMDB=none go mod download -json github.com/homeport/dyff@v1.5.7
{
        "Path": "github.com/homeport/dyff",
        "Version": "v1.5.7",
        "Error": "github.com/homeport/dyff@v1.5.7: verifying go.mod: invalid GOSUMDB: malformed verifier id",
        "Info": "/home/matt/go/1.22.1/pkg/mod/cache/download/github.com/homeport/dyff/@v/v1.5.7.info",
        "Origin": {
                "VCS": "git",
                "URL": "https://github.com/homeport/dyff",
                "Hash": "86654f8e002a69fdabc87ac954f7d0fa11e6b84b",
                "Ref": "refs/tags/v1.5.7"
        }
}

That error comes from here.

Now for the real evidence:

❯ git clone git@github.com:homeport/dyff.git
Cloning into 'dyff'...
remote: Enumerating objects: 5069, done.
remote: Counting objects: 100% (774/774), done.
remote: Compressing objects: 100% (289/289), done.
remote: Total 5069 (delta 656), reused 485 (delta 485), pack-reused 4295
Receiving objects: 100% (5069/5069), 7.39 MiB | 9.38 MiB/s, done.
Resolving deltas: 100% (3157/3157), done.

/tmp took 2s
❯ cd dyff

dyff on  main via 🐹 v1.22.1
❯ git checkout tags/v1.5.7
Note: switching to 'tags/v1.5.7'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 86654f8 Remove `snap` target for now

dyff (86654f8) via 🐹 v1.22.1
❯ ls
assets  cmd  go.mod  go.sum  internal  LICENSE  Makefile  pkg  README.md  scripts

dyff (86654f8) via 🐹 v1.22.1
❯ go-checksum . github.com/homeport/dyff@v1.5.7
directory: .
{
        "HashSynthesized": "9b038647867396be3c4686af75e06e0e25691f63c854a303d7dc6ebe2acc0c41",
        "HashSynthesizedBase64": "mwOGR4Zzlr48RoavdeBuDiVpH2PIVKMD19xuvirMDEE=",
        "GoCheckSum": "h1:mwOGR4Zzlr48RoavdeBuDiVpH2PIVKMD19xuvirMDEE="
}

mwOGR4Zzlr48RoavdeBuDiVpH2PIVKMD19xuvirMDEE is the checksum the Go binary calculated and gave to Athens. The checksum in proxy.golang.org and sum.golang.org works but is different from what's checked into the repository at that tag, hence it fails the checksum.

matt0x6F · 2024-04-17T04:11:28Z

To be clear, this is not a bug with Athens. Athens did what it is supposed to do and what I said I originally suspected is likely what happened: The repository maintainer reassigned that tag to a different commit but it was already in proxy.golang.org and sum.golang.org.

One way to mitigate this is to tell Athens to use proxy.golang.org:
GoBinaryEnvVars = ["GOPROXY=proxy.golang.org,direct"]

matt0x6F · 2024-04-17T04:18:07Z

Lastly, the smoking gun. The Checksum in sum.golang.org comes from this commit

dyff (a901b82) via 🐹 v1.22.1
❯ git checkout e2f30e959f2105954276e58a05c1f3526c1d0c0d
Previous HEAD position was a901b82 Bump `golang.org/x` dependencies
HEAD is now at e2f30e9 Enable Windows builds (again)

dyff (e2f30e9) via 🐹 v1.22.1
❯ go-checksum . github.com/homeport/dyff@v1.5.7
directory: .
{
        "HashSynthesized": "db822b75201bd9fa6b47aed71cc21a806a1cd5ce1534157d89b3452880ac4a2b",
        "HashSynthesizedBase64": "24IrdSAb2fprR67XHMIagGoc1c4VNBV9ibNFKICsSis=",
        "GoCheckSum": "h1:24IrdSAb2fprR67XHMIagGoc1c4VNBV9ibNFKICsSis="
}

matt0x6F added the bug Something isn't working label Apr 16, 2024

matt0x6F added this to the 0.13.4 milestone Apr 16, 2024

matt0x6F changed the title ~~verifying module: checksum mismatch but only with proxy~~ Athens does not calculate the correct checksum Apr 16, 2024

matt0x6F removed the bug Something isn't working label Apr 17, 2024

matt0x6F closed this as completed Apr 17, 2024

matt0x6F modified the milestones: 0.13.4, 0.14.0 Apr 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Athens does not calculate the correct checksum #1881

Athens does not calculate the correct checksum #1881

idefixcert commented Sep 5, 2023

bozaro commented Sep 27, 2023 •

edited

Loading

matt0x6F commented Apr 15, 2024

bozaro commented Apr 15, 2024 •

edited

Loading

matt0x6F commented Apr 15, 2024 •

edited

Loading

matt0x6F commented Apr 15, 2024

matt0x6F commented Apr 16, 2024

matt0x6F commented Apr 16, 2024

matt0x6F commented Apr 16, 2024

matt0x6F commented Apr 16, 2024

matt0x6F commented Apr 17, 2024 •

edited

Loading

matt0x6F commented Apr 17, 2024

matt0x6F commented Apr 17, 2024

matt0x6F commented Apr 17, 2024

Athens does not calculate the correct checksum #1881

Athens does not calculate the correct checksum #1881

Comments

idefixcert commented Sep 5, 2023

bozaro commented Sep 27, 2023 • edited Loading

matt0x6F commented Apr 15, 2024

bozaro commented Apr 15, 2024 • edited Loading

matt0x6F commented Apr 15, 2024 • edited Loading

matt0x6F commented Apr 15, 2024

matt0x6F commented Apr 16, 2024

matt0x6F commented Apr 16, 2024

matt0x6F commented Apr 16, 2024

matt0x6F commented Apr 16, 2024

matt0x6F commented Apr 17, 2024 • edited Loading

matt0x6F commented Apr 17, 2024

matt0x6F commented Apr 17, 2024

matt0x6F commented Apr 17, 2024

bozaro commented Sep 27, 2023 •

edited

Loading

bozaro commented Apr 15, 2024 •

edited

Loading

matt0x6F commented Apr 15, 2024 •

edited

Loading

matt0x6F commented Apr 17, 2024 •

edited

Loading