Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: checksum mismatch for x8s.io/client-go@v8.0.0+incompatible #27925

Closed
jharshman opened this issue Sep 28, 2018 · 40 comments

Comments

Projects
None yet
@jharshman
Copy link

commented Sep 28, 2018

What version of Go are you using (go version)?

Go 1.11

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

darwin
amd64

What did you do?

go build ./...
reproduction is difficult because including the package k8s.io/client-go causes incompatible dependencies to be pulled in which is something I have to solve manually.

What did you expect to see?

A repeat of successful builds from the day before.

What did you see instead?

A checksum mismatch this morning after cloning the project to my personal laptop and attempting a build.

go: verifying k8s.io/client-go@v8.0.0+incompatible: checksum mismatch
        downloaded: h1:7Zl+OVXn0bobcsi4NEZGdoQDTE9ij1zPMfM21+yqQsM=
        go.sum:     h1:2pUaSg2x6iEHr8cia6zmWhoCXG1EDG9TCx9s//Aq7HY=

@jharshman jharshman changed the title Go Modules not producing reproducable builds. Go mod not producing reproducible builds. Sep 28, 2018

@jharshman

This comment has been minimized.

Copy link
Author

commented Sep 28, 2018

Okay so the workaround is to manually go get the correct version and then run the build again. This is a little unpredictable IMO. I'm guessing this stuff will clear up as more packages add module support and semantic versioning.

@katiehockman

This comment has been minimized.

Copy link
Contributor

commented Sep 28, 2018

@jharshman is there further investigation that needs to be done for this issue? Is there documentation that you are suggesting should be updated? If not, it sounds like this issue can be closed.

@bcmills bcmills changed the title Go mod not producing reproducible builds. cmd/go: checksum mismatch for x8s.io/client-go@v8.0.0+incompatible Sep 28, 2018

@bcmills bcmills added the modules label Sep 28, 2018

@bcmills bcmills added this to the Go1.12 milestone Sep 28, 2018

@bcmills

This comment has been minimized.

Copy link
Member

commented Sep 28, 2018

+incompatible versions should not cause checksum mismatches.

Is it possible that you were running more than one build concurrently? (#26794)

@jharshman

This comment has been minimized.

Copy link
Author

commented Sep 28, 2018

@katiehockman I would say this particular issue shouldn't be closed because this seems to be a symptom of a larger issue and needs to be addressed. As @bcmills mentioned, +incompatible versions should not cause checksum mismatches but in this case it has.

And @bcmills to answer your question, no this was the only build running. There were no other builds going on concurrently.

@F21

This comment has been minimized.

Copy link

commented Oct 1, 2018

Also seeing this problem but not sure if this is due to GOOS mismatch. I am using WSL (Ubuntu x64) on Windows 10 to run go mod tidy and generate the corresponding go.sum file. I am building and testing using Gitlab CI using a Go 1.11 Alpine x64 container and I am seeing:

go: verifying k8s.io/client-go@v8.0.0+incompatible: checksum mismatch
	downloaded: h1:7Zl+OVXn0bobcsi4NEZGdoQDTE9ij1zPMfM21+yqQsM=
	go.sum:     h1:2pUaSg2x6iEHr8cia6zmWhoCXG1EDG9TCx9s//Aq7HY=

go version on both the ci machine and my own machine outputs: go version go1.11 linux/amd64.

I also recreate the go.sum and go.mod files using an Go 1.11 alpine container (the same one used in CI) and still see the same problem.

@jharshman

This comment has been minimized.

Copy link
Author

commented Oct 4, 2018

@katiehockman please remove the WaitingForInfo label unless there is more information that you are requesting.
@bcmills is there a solution in the pipe for this? It's very reproducible so I can image other users are experiencing it as well.

The best solution I have found is to go mod vendor once you have a working build.

@bcmills

This comment has been minimized.

Copy link
Member

commented Oct 4, 2018

@jharshman Could you try a Go build from source at commit 1bca6ce and see if it still reproduces?

I wonder if this is a side-effect of #27153.

@leakingtapan

This comment has been minimized.

Copy link

commented Oct 5, 2018

Im having similar issue with golang master branch on my travis build. What's resolution for this?

@bcmills

This comment has been minimized.

Copy link
Member

commented Oct 5, 2018

@leakingtapan, try a go built from source at the above-mentioned commit, and let us know whether it helps.

For CI-related issues, though, you're more likely running up against #26794.

@leakingtapan

This comment has been minimized.

Copy link

commented Oct 5, 2018

I tried 1bca6ce with no luck. I also tried 8081862 which was working on my Travis build, but now even this is failing.

go: verifying k8s.io/kubernetes@v1.11.2: checksum mismatch
        downloaded: h1:AAwZncqPvpeV6DPF8/z2g05/79cumA6IunGDByI1bOs=
        go.sum:     h1:2/lmzYbN17Mr23mX/p4ODMm/MBWTeu2Q1Bvsw82YC40=
@cosmonawt

This comment has been minimized.

Copy link

commented Oct 8, 2018

Same issue here:

go: downloading k8s.io/client-go v9.0.0+incompatible
go: downloading k8s.io/api v0.0.0-20181006043042-ded30614edd7
go: verifying k8s.io/client-go@v9.0.0+incompatible: checksum mismatch
        downloaded: h1:/PdJjifJTjMFe0G4ESclZDcwF1+bFePTJ2xf+MXjcvs=
        go.sum:     h1:NXWpDuPFeVB5lYP1fTqJUtwigjtmRXJNtndnN53ldGI=
@jasonkeene

This comment has been minimized.

Copy link
Contributor

commented Oct 9, 2018

@leakingtapan @cosmonawt @jharshman @F21

If you are running 1bca6ce your h1 hashes will, unfortunately, have to be updated once. Both kubernetes/kubernetes and kubernetes/client-go use export-subst which is disabled by this change:

If you are running 1bca6ce you should get the same hash moving forward. Just delete the old hash and use the new one. Again, you should only have to do this once.

I imagine a lot of folks will run into this issue since k8s repos appear to use export-subst in various places.

If you are seeing inconsistent hashing results I recommend:

  1. Build go from source after 1bca6ce
  2. Delete the old h1 hash for k8s.io/kubernetes k8s.io/client-go and any other repo using export-subst.
  3. Build your project and record the new hash for k8s.io/kubernetes k8s.io/client-go etc

If the hash changes for you after that then please report back.

M00nF1sh added a commit to M00nF1sh/aws-alb-ingress-controller that referenced this issue Oct 9, 2018

@leakingtapan

This comment has been minimized.

Copy link

commented Oct 9, 2018

@jasonkeene thx for providing the solution. In order to not break people who is using go1.11 to build our package, I decided to stay with go1.11 for a while. This makes my travis build and mac build happy. However, when I tried build on a linux machine the issue even happen on go1.11, which is surprising. Here is the output:

mac

>> go version
go version go1.11 darwin/amd64

>> echo $GO111MODULE
on

>> make
make
mkdir -p bin
go build -o bin/aws-ebs-csi-driver ./cmd/aws-ebs-csi-driver

linux

>> go version
go version go1.11 linux/amd64

>> echo $GO111MODULE                                                                                                                                                                                                            
on

>> make
mkdir -p bin
go build -o bin/aws-ebs-csi-driver ./cmd/aws-ebs-csi-driver
go: verifying k8s.io/kubernetes@v1.11.2: checksum mismatch
        downloaded: h1:AAwZncqPvpeV6DPF8/z2g05/79cumA6IunGDByI1bOs=
        go.sum:     h1:2/lmzYbN17Mr23mX/p4ODMm/MBWTeu2Q1Bvsw82YC40=
make: *** [aws-ebs-csi-driver] Error 1
@jasonkeene

This comment has been minimized.

Copy link
Contributor

commented Oct 9, 2018

Yes this might be the problem 1bca6ce was intended to solve. Can you diff the contents of the two zip files that produce the different hashes?

@jharshman

This comment has been minimized.

Copy link
Author

commented Oct 10, 2018

Thanks for the solution @jasonkeene , but building Go from source is the verification, not a long term solution. We need to be able to build projects reliably cross teams, platforms, and in CI.

It sounds like a bug-fix release needs to be cut from this.

@jasonkeene

This comment has been minimized.

Copy link
Contributor

commented Oct 10, 2018

@jharshman For sure compiling from source was just to test to make sure this was your issue.

As far as the bug fix release, pretty sure @bcmills is on it here: #28094

I would still like you to try the above steps to make sure you are not running into another issue and that the bug fix will actually fix your problem.

automaticserver pushed a commit to automaticserver/lxe that referenced this issue Oct 11, 2018

temporarely leave go.sum out because of checksum mismatch bug, will b…
…e enabled again once release containing this fix is out. See golang/go#27925
@jharshman

This comment has been minimized.

Copy link
Author

commented Oct 11, 2018

@jasonkeene sure let me verify that I don't hit any other issues 👍

@jsternberg

This comment has been minimized.

Copy link

commented Oct 15, 2018

I'm not sure if it's related, but this may be related to an issue we've been having with modules: #27952

We've had situations where the incomplete project didn't compile, but we've also had times that it compiled, but had the wrong hash because files were missing.

@jharshman

This comment has been minimized.

Copy link
Author

commented Oct 16, 2018

@jasonkeene verified the fix by compiling from source after 1bca6ce. Works after removing the older hashes.

@lastzero

This comment has been minimized.

Copy link

commented Dec 18, 2018

Great to see this is fixed! Any idea when it will be released? Go 1.11.4 is still affected. I've just wasted half a day because GoLand on OS X is updating the sum file automatically with checksums that don't work in Linux containers. Turning this off at least makes it possible to compile again, but without module support / code completion.

OS X (Go 1.11.4, Git 2.17.2)
github.com/kylelemons/go-gypsy v0.0.0-20160905020020-08cad365cd28 h1:SzHLOWB819X9aUXh2ct6krDIoIujoTsj5J1rDMjkRyo=

Linux (Go 1.11.4, Git 2.17.1)
github.com/kylelemons/go-gypsy v0.0.0-20160905020020-08cad365cd28 h1:mkl3tvPHIuPaWsLtmHTybJeoVEW7cbePK73Ir8VtruA=

@simonpasquier

This comment has been minimized.

Copy link

commented Dec 18, 2018

@lastzero you're most probably hitting #29278 (comment)

@lastzero

This comment has been minimized.

Copy link

commented Dec 18, 2018

@simonpasquier Thank you! Looks like the follow-up "bug". For those also affected: I had to change settings in GoLand (didn't automatically detect latest Go version) plus delete Go module caches locally and on Travis CI.

@irvifa

This comment has been minimized.

Copy link

commented Feb 7, 2019

Using 1.11.5 and still have this impact

OS: Darwin/Mojave

Edit: apparently it works after I do:

rm go.sum
go mod vendor
@int128

This comment has been minimized.

Copy link

commented Feb 28, 2019

If your machine is macOS or WSL and your CI is Linux, you can add this command to the CI config.

sed -e '/^k8s.io\/client-go /d' -i go.sum

Here is an example for CircleCI:

version: 2
jobs:
  build:
    docker:
      - image: circleci/golang:1.11.2
    steps:
      - checkout
      # workaround for https://github.com/golang/go/issues/27925
      - run: sed -e '/^k8s.io\/client-go /d' -i go.sum
      - run: go vet
@runner-mei

This comment has been minimized.

Copy link

commented Apr 26, 2019

d:\xxx>go version
go version go1.12.4 windows/amd64

d:\xxx>go mod tidy
go: downloading github.com/kevinburke/go-bindata v3.11.0+incompatible
verifying github.com/kevinburke/go-bindata@v3.11.0+incompatible: checksum mismatch
downloaded: h1:RcC+GJNmrBHbGaOpQ9MBD8z22rdzlIm0esDRDkyxd4s=
go.sum: h1:GiPs9jxaG2xY1B5Dt/d/yHUOMlTk14uS35VcmHrdo4I=

@runner-mei

This comment has been minimized.

Copy link

commented Apr 26, 2019

verifying github.com/kylelemons/go-gypsy@v0.0.0-20160905020020-08cad365cd28: che
cksum mismatch
downloaded: h1:mkl3tvPHIuPaWsLtmHTybJeoVEW7cbePK73Ir8VtruA=
go.sum: h1:SzHLOWB819X9aUXh2ct6krDIoIujoTsj5J1rDMjkRyo=

@lastzero

This comment has been minimized.

Copy link

commented Apr 26, 2019

@runner-mei Did you try go clean -modcache?

@runner-mei

This comment has been minimized.

Copy link

commented Apr 26, 2019

@lastzero oooo, it is ok!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.