Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go.sum checksum different on Windows vs. Linux #33178

Closed
kjk opened this issue Jul 19, 2019 · 9 comments
Closed

cmd/go: go.sum checksum different on Windows vs. Linux #33178

kjk opened this issue Jul 19, 2019 · 9 comments

Comments

@kjk
Copy link

@kjk kjk commented Jul 19, 2019

What version of Go are you using (go version)?

$ go version
go version go1.12.7 windows/amd64

$go version
go version go1.12.4 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\kjk\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\kjk\go
set GOPROXY=
set GORACE=
set GOROOT=c:\go
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\Users\kjk\src\apps\offdocs\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\kjk\AppData\Local\Temp\go-build015649226=/tmp/go-build -gno-record-gcc-switches

What did you do?

I have a program that imports github.com/alecthomas/chroma. I've created go.sum on Windows (by building a program with go 1.2.7) and checked it in.

I checked it out on Linux (Ubuntu 18.04) and tried to build it with go 1.2.4.

What did you expect to see?

It should work.

What did you see instead?

It failed with:

verifying github.com/alecthomas/chroma@v0.6.5: checksum mismatch
        downloaded: h1:1gZ11dfxovHkbAM+xzyxjgxhG3Gr7U/cfOXI2eoEBkU=
        go.sum:     h1:4CN9fmMfN/k6Ppb+djxJqu3BG7WuLkct0dvScOc0w2w=

I fixed it by deleting go.sum but after a build it was indeed different on Linux than on Windows:

-github.com/alecthomas/chroma v0.6.5 h1:4CN9fmMfN/k6Ppb+djxJqu3BG7WuLkct0dvScOc0w2w=
+github.com/alecthomas/chroma v0.6.5 h1:1gZ11dfxovHkbAM+xzyxjgxhG3Gr7U/cfOXI2eoEBkU=

Please note I know little about how go.sum works but here's my theory:

  • go build calculates checksum in go.sum as hash of file content
  • go build downloads repositories using git installed on the system
  • git can change newlines depending on core.autocrlf settings, changing the content and therefore changing the hash

On Windows I have:

PS C:\Users\kjk\src\apps\offdocs> git config --list
core.autocrlf=true

On Linux I have:

kjk@kjk-thinkpad25:~/src/onlinetools$ git config --list
core.autocrlf=false

Presumably, we would prefer go.sum not be affected by local value of git's core.autocrlf setting.

I imagine it would be possible for go tool to invoke git in such a way as to over-ride local setting and ensure that core.autocrlf=false when downloading dependent repos locally for module cache.

@dmitshur dmitshur added this to the Go1.14 milestone Jul 19, 2019
@dmitshur dmitshur changed the title go.sum checksum different on Windows vs. Linux cmd/go: go.sum checksum different on Windows vs. Linux Jul 19, 2019
@kjk

This comment has been minimized.

Copy link
Author

@kjk kjk commented Jul 19, 2019

Or it could be a stale module cache / hash generated with earlier go version.
Apparently there was a change in how hash was calculated.
I failed to create a reproducible by trying to force this. I created:

main.go:

package main

import (
	"fmt"
	_ "github.com/alecthomas/chroma"
)

func main() {
	fmt.Printf("Hello\n")
}

go.mod:

module tst

go 1.12

require github.com/alecthomas/chroma v0.6.5

And a repro.bat:

del /s /q go.sum
git config --unset-all core.autocrlf

git config --add core.autocrlf  true
git config --get core.autocrlf

go clean -modcache
go build .
copy go.sum go.sum1

del /s /q go.sum

git config --unset-all core.autocrlf
git config --add core.autocrlf false
git config --get core.autocrlf

go clean -modcache
go build .
copy go.sum go.sum2

The theory was to re-create go.sum with core.autocrlf=true and core.autocrlf=false but it seems in both cases I get the same:

github.com/alecthomas/chroma v0.6.5 h1:1gZ11dfxovHkbAM+xzyxjgxhG3Gr7U/cfOXI2eoEBkU=

But maybe I'm just not reproducing whatever git settings that led to creating:

github.com/alecthomas/chroma v0.6.5 h1:4CN9fmMfN/k6Ppb+djxJqu3BG7WuLkct0dvScOc0w2w=

One way to confirm the "format change" theory would be to rerun this with a version of go before the go.sum format change and see what hash it produces for github.com/alecthomas/chroma.

I don't know when this change happened.

@jayconrod

This comment has been minimized.

Copy link
Contributor

@jayconrod jayconrod commented Jul 19, 2019

There was a change in the way sums were calculated between go1.11.3 and go1.11.4 (we will ensure that never happens again). Is it possible the h1:4CN9... sum or the module it was derived from was from before that? Is there any way to reproduce this after cleaning the cache (go clean -modcache)?

The checksum algorithm explicitly sets core.autocrlf to avoid problems with line endings, so that shouldn't be an issue.

For the record, h1:1gZ11dfxovHkbAM+xzyxjgxhG3Gr7U/cfOXI2eoEBkU= is the correct sum (per sum.golang.org).

@kjk

This comment has been minimized.

Copy link
Author

@kjk kjk commented Jul 19, 2019

It's possible. I imagine running go build with go <1.11.3 on the sample program in #33178 (comment) and checking the checksum would resolve this for sure.

I don't have a setup with go 1.11.3 thought.

@FiloSottile

This comment has been minimized.

Copy link
Member

@FiloSottile FiloSottile commented Jul 20, 2019

You can get Go 1.11.3 with go get golang.org/dl/go1.11.3 alongside your system Go.

@kjk

This comment has been minimized.

Copy link
Author

@kjk kjk commented Jul 20, 2019

It's still h1:1gZ11dfxovHkbAM+xzyxjgxhG3Gr7U/cfOXI2eoEBkU= with go 1.11 (as best as I can tell).

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Jul 22, 2019

There was an earlier extraction bug (#27153) fixed in Go 1.11.2, and an even earlier one way back in the vgo phase (#26229). Any idea how old the module cache on your Windows machine is?

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Jul 22, 2019

Also note that before Go 1.12, we had no locking or synchronization on the module cache. Is it possible that the cache entry for this module was corrupted due to concurrent go commands?

@kjk

This comment has been minimized.

Copy link
Author

@kjk kjk commented Jul 22, 2019

Yes, it's possible it comes from corrupted cache.

I've ran go clean -modcache since so no idea when that cache was corrupted.

It's fine to close this bug as I see no way to make progress determining the root cause.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Jul 22, 2019

Thanks. Do let us know if you see this again going forward, especially if you discover a clue as to the underlying cause.

@bcmills bcmills closed this Jul 22, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.