Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: git remote (Gerrit) did not send all necessary objects #31399

Closed
ukai opened this issue Apr 11, 2019 · 61 comments
Closed

cmd/go: git remote (Gerrit) did not send all necessary objects #31399

ukai opened this issue Apr 11, 2019 · 61 comments
Assignees
Labels
Milestone

Comments

@ukai
Copy link
Contributor

@ukai ukai commented Apr 11, 2019

What version of Go are you using (go version)?

$ go version
go version go1.12 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/usr/local/google/home/ukai/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/usr/local/google/home/ukai/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/google-golang"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/google-golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build741696436=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ ls /tmp/go
ls: cannot access '/tmp/go': No such file or directory
$ mkdir /tmp/go
$ GOPATH=/tmp/go GO111MODULE=on go install go.chromium.org/goma/server/cmd/remoteexec_proxy

What did you expect to see?

go modules downloaded and binary is built.

What did you see instead?

...
go: finding grpc.go4.org v0.0.0-20170609214715-11d0a25b4919
go: google.golang.org/api@v0.0.0-20181030000543-1d582fd0359e: git -c protocol.version=0 fetch --unshallow -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /tmp/go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0: exit status 128:
        fatal: bad object c7964aab01c00a07d0cc6216a78a5570459cebfd
        error: remote did not send all necessary objects
..
go: error loading module requirements

I could fix it by

$ cd /tmp/go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0; git fetch --unshallow

then go install works as expected.

https://bugs.chromium.org/p/chromium/issues/detail?id=951546

@agnivade

This comment has been minimized.

Copy link
Contributor

@agnivade agnivade commented Apr 11, 2019

@KillerX

This comment has been minimized.

Copy link

@KillerX KillerX commented Apr 11, 2019

I can confirm that this is happening to me as well. It is not 100% constant. My docker builds experience it about 98% of the time.

go: google.golang.org/api@v0.0.0-20180910000450-7ca32eb868bf: git -c protocol.version=0 fetch --unshallow -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0: exit status 1:
	fatal: bad object c7964aab01c00a07d0cc6216a78a5570459cebfd
	error: https://code.googlesource.com/google-api-go-client did not send all necessary objects

A git fetch seems to fix it.

Same issues with go 1.11.5, 1.11.8, 1.12.3 in Docker (golang:1.12.3), and installed locally on Ubuntu.

Local git:

➜   git --version 
git version 2.19.1

I also less frequently get this issue:

go: cloud.google.com/go@v0.0.0-20180131234750-2de512d2700d: git -c protocol.version=0 fetch --unshallow -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/b0e27935eb83c1d7843713bafab507e95768b550f0552cb42d9f41e5fd9c8375: exit status 1:
        fatal: bad object 3fdddcac5af93820a98679aa0008db7413a13265
        error: https://code.googlesource.com/gocloud did not send all necessary objects

EDIT:

It appears that this patch resolves the issue for my builds:

--- a/packages/go/packagename/go.mod
+++ b/packages/go/packagename/go.mod
@@ -7,7 +7,7 @@ require (
        redacted/logging v0.0.0
        golang.org/x/net v0.0.0-20190320064053-1272bf9dcd53 // indirect
        golang.org/x/oauth2 v0.0.0-20190319182350-c85d3e98c914 // indirect
-       google.golang.org/api v0.3.1
+       google.golang.org/api  v0.0.0-20180603000442-8e296ef26005
        google.golang.org/genproto v0.0.0-20190404172233-64821d5d2107
        google.golang.org/grpc v1.19.1
 )
@jayconrod jayconrod changed the title cmd/go: error loading module requirements cmd/go: git remote did not send all necessary objects Apr 11, 2019
@jayconrod jayconrod changed the title cmd/go: git remote did not send all necessary objects cmd/go: git remote (Gerrit) did not send all necessary objects Apr 11, 2019
@jayconrod

This comment has been minimized.

Copy link
Contributor

@jayconrod jayconrod commented Apr 11, 2019

Reproduced locally with the same error.

Also seen with the same instructions as above:

go: golang.org/x/exp@v0.0.0-20190121172915-509febef88a4: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /var/folders/rq/x0692kqj6ml8cvrhcqh5bswc008xj1/T/tmp.yf9bsDvK/pkg/mod/cache/vcs/abc0aaaae9fbb1a906994bfb84d444789fc395ac48e2b740ecbbfeebd3047a32: exit status 128:
	fatal: remote error: authentication error

Perhaps Gerrit is acting up again?

cc @heschik @andybons @hakim @dmitshur

@jayconrod jayconrod self-assigned this Apr 11, 2019
@jayconrod

This comment has been minimized.

Copy link
Contributor

@jayconrod jayconrod commented Apr 11, 2019

I've traced this to a sequence of Git commands that reproduces this issue outside of Go. The Gerrit team has been notified and is working to resolve the issue.

@mr-tron

This comment has been minimized.

Copy link

@mr-tron mr-tron commented Apr 11, 2019

Is it some workaround for the problem?

@heschik

This comment has been minimized.

Copy link
Contributor

@heschik heschik commented Apr 11, 2019

The latest we have from the Gerrit team is that this might be a bug specific to the content of the google.golang.org/api repository. As suggested above, a git fetch should work around it. I think fetching a different version might help too.

@mr-tron

This comment has been minimized.

Copy link

@mr-tron mr-tron commented Apr 11, 2019

Yes. Second running also helps, but it very ugly hack. I added vendor dir to my projects. It looks like better workaround.

@draftcode

This comment has been minimized.

Copy link

@draftcode draftcode commented Apr 11, 2019

Hello, this is the team that manages googlesource.com. *.googlesource.com is hosted by using JGit, and we believe that there's a bug in upstream JGit that cannot handle unshallow git-fetch. We recognize that this is reproducible easily, but we cannot spot the buggy code in the JGit. If there's an option to do a full clone always instead of partial clone, that can be a workaround.

@draftcode

This comment has been minimized.

Copy link

@draftcode draftcode commented Apr 12, 2019

We identified the root cause and the fix is uploaded at https://git.eclipse.org/r/c/140463/. Our in-house JGit comitters are already gone home, so this will be reviewed tomorrow at best. If everything goes well and our SREs are happy about a Friday release, we might be able to cherry-pick this fix tomorrow. The worst case ETA would be next week.

@jrn

This comment has been minimized.

Copy link

@jrn jrn commented Apr 12, 2019

https://git.eclipse.org/r/c/140463/

That did the trick. If it happens again, we want to know.

jim added a commit to transcom/mymove that referenced this issue Apr 12, 2019
Waiting on the following to be resolved: golang/go#31399
@jim jim mentioned this issue Apr 12, 2019
0 of 1 task complete
@chmking

This comment has been minimized.

Copy link

@chmking chmking commented Apr 12, 2019

@jrn @draftcode

Everything was working this morning but I've seen a regression in the past 10 mins:

12-Apr-2019 13:21:54 | go: google.golang.org/api@v0.0.0-20181021000519-a2651947f503: git -c protocol.version=0 fetch --unshallow -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0: exit status 1:
12-Apr-2019 13:21:54 | fatal: bad object 19e022d8cf43ce81f046bae8cc18c5397cc7732f
12-Apr-2019 13:21:54 | error: https://code.googlesource.com/google-api-go-client did not send all necessary objects
12-Apr-2019 13:27:12 | verifying google.golang.org/api@v0.3.1/go.mod: checksum mismatch
12-Apr-2019 13:27:12 | downloaded: h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0=
12-Apr-2019 13:27:12 | go.sum:     h1:6wY9I6uQWHQ8EM57III9mq/AjF+i8G65rmVagqKMtkk=
@mr-tron

This comment has been minimized.

Copy link

@mr-tron mr-tron commented Apr 12, 2019

Still doesnt work for me.

@jayconrod

This comment has been minimized.

Copy link
Contributor

@jayconrod jayconrod commented Apr 12, 2019

I've tried to reproduce this again, and it's now working for me.

@chmking @mr-tron Are you able to reproduce this with a clean module cache using the original instructions at the top of the issue? Could you post the output of go install -x go.chromium.org/goma/server/cmd/remoteexec_proxy with the failure? The -x flag should show the git commands run in each vcs cache, so that will help us isolate the issue.

@chmking

This comment has been minimized.

Copy link

@chmking chmking commented Apr 12, 2019

@jayconrod This is happening on our CI machines running the build in a docker without a go mod cache. It worked this morning without fail many times until it started failing at the time of my post. Now the issues is intermittent.

I can try to replicate it for you later.

@draveness

This comment has been minimized.

Copy link
Contributor

@draveness draveness commented Apr 14, 2019

I met the same problem when using go get kubernetes/client-go#590

$ go get -u -v k8s.io/client-go
Fetching https://cloud.google.com/go?go-get=1
Parsing meta tags from https://cloud.google.com/go?go-get=1 (status code 200)
get "cloud.google.com/go": found meta tag get.metaImport{Prefix:"cloud.google.com/go", VCS:"git", RepoRoot:"https://code.googlesource.com/gocloud"} at https://cloud.google.com/go?go-get=1
go: finding cloud.google.com/go v0.0.0-20160913182117-3b1ae45394a2
go: cloud.google.com/go@v0.0.0-20160913182117-3b1ae45394a2: git -c protocol.version=0 fetch --unshallow -f https://code.googlesource.com/gocloud refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /Users/draveness/go/pkg/mod/cache/vcs/b0e27935eb83c1d7843713bafab507e95768b550f0552cb42d9f41e5fd9c8375: exit status 128:
	fatal: bad object a5913b3f7deecba45e98ff33cefbac4fd204ddd7
	error: remote did not send all necessary objects
go: error loading module requirements

$ git --version
git version 2.21.0
@hazcod

This comment has been minimized.

Copy link

@hazcod hazcod commented Apr 15, 2019

Is there a workaround for go get? I am unable to develop.
Edit: I skipped over the workaround.

@agnivade

This comment has been minimized.

Copy link
Contributor

@agnivade agnivade commented Apr 15, 2019

Please see the first comment for the workaround.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Apr 19, 2019

@draftcode, is someone on the Gerrit side looking into this?

@bcmills bcmills added this to the Unreleased milestone Apr 19, 2019
@draftcode

This comment has been minimized.

Copy link

@draftcode draftcode commented Apr 19, 2019

Reproduced with JGit:

git init . --bare
git remote add origin git://localhost:9418/google-api-go-client.git
git fetch -f --depth=1 origin refs/tags/v0.3.2:refs/tags/v0.3.2
git fetch -f --depth=1 origin refs/tags/v0.1.0:refs/tags/v0.1.0
git fetch -f --depth=1 origin refs/tags/v0.3.1:refs/tags/v0.3.1
git fetch -f --unshallow origin 'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*'
@draftcode

This comment has been minimized.

Copy link

@draftcode draftcode commented Apr 19, 2019

I found another bug in JGit code that made it exclude dcbf3e3c23816703a8f5ebfe6583cc3fc2793526. But after I fix that, I found another bug that doesn't send a parent of v0.1.0. I'm debugging JGit more.

@draftcode

This comment has been minimized.

Copy link

@draftcode draftcode commented Apr 19, 2019

https://git.eclipse.org/r/c/140900/ and https://git.eclipse.org/r/c/140901/ will fix the the three tag unshallow case. Tried all three previously reproduced cases.

@iamtakingiteasy

This comment has been minimized.

Copy link

@iamtakingiteasy iamtakingiteasy commented Apr 22, 2019

A possible temporary workaround for docker builds might be

RUN ( \
      export GO111MODULE=on && cd / && go get google.golang.org/api && \
      cd "$(grep -l https://code.googlesource.com/google-api-go-client $GOPATH/pkg/mod/cache/vcs/*.info | sed 's/\.info$//')" && \
      git fetch -f https://code.googlesource.com/google-api-go-client 'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*' \
    )
@jayconrod

This comment has been minimized.

Copy link
Contributor

@jayconrod jayconrod commented Apr 22, 2019

Still seems to be happening. Pinged internal bug.

@draftcode

This comment has been minimized.

Copy link

@draftcode draftcode commented Apr 22, 2019

#31399 (comment) is not rolled out yet.

@jayconrod

This comment has been minimized.

Copy link
Contributor

@jayconrod jayconrod commented Apr 22, 2019

Oops, I thought those CLs had been deployed already.

@draftcode Do you expect this will fix the GOPATH=/tmp/go GO111MODULE=on go install go.chromium.org/goma/server/cmd/remoteexec_proxy command from the original example?

@anasanzari

This comment has been minimized.

Copy link

@anasanzari anasanzari commented Apr 23, 2019

@iamtakingiteasy not working anymore

@iamtakingiteasy

This comment has been minimized.

Copy link

@iamtakingiteasy iamtakingiteasy commented Apr 23, 2019

@anasanzari true, but it seems original issue of fetching https://code.googlesource.com/google-api-go-client from go mod tools was somehow fixed despite commits not being merged (or at least I couldn't find a way to reproduce), this workaround is not needed anymore.

You can remove it from your pipeline and build as normal if that was your use-case.

Also it is indeed odd such fetch would not work. Might be somewhat related issue (which those commits might fix).

@jayconrod

This comment has been minimized.

Copy link
Contributor

@jayconrod jayconrod commented Apr 23, 2019

All fixes are rolled out now, and this seems to be working. Thanks @draftcode for working on this!

I'll close this but everyone please comment if you're still seeing this, and we'll investigate.

@jayconrod jayconrod closed this Apr 23, 2019
@anasanzari

This comment has been minimized.

Copy link

@anasanzari anasanzari commented Apr 24, 2019

@jayconrod issue is still there. I've added my case in https://github.com/anasanzari/golang_build_check.
docker build . fails with the same error.

@bradfitz

This comment has been minimized.

Copy link
Contributor

@bradfitz bradfitz commented Apr 24, 2019

@anasanzari, thanks. I can reproduce the failure.

Gerrit/jgit is still broken:

go: google.golang.org/api@v0.0.0-20181030000543-1d582fd0359e: git -c protocol.version=0 fetch --unshallow -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0: exit status 128:
        fatal: missing tree object 'dcbf3e3c23816703a8f5ebfe6583cc3fc2793526'
        error: remote did not send all necessary objects
go: error loading module requirements

Reopening.

@bradfitz bradfitz reopened this Apr 24, 2019
@bradfitz

This comment has been minimized.

Copy link
Contributor

@bradfitz bradfitz commented Apr 24, 2019

@draftcode, please investigate?

@draftcode

This comment has been minimized.

Copy link

@draftcode draftcode commented Apr 24, 2019

I can reproduce this, but I cannot reproduce this with JGit.

@anasanzari

This comment has been minimized.

Copy link

@anasanzari anasanzari commented Apr 25, 2019

can someone give a quick fix for this?

@tparker

This comment has been minimized.

Copy link

@tparker tparker commented Apr 25, 2019

The reproduction case above only failed in production-like environments, so it took longer to track down. The next fix forgoes potential data transfer optimizations in the name of correctness. Unshallow operations will be full clones. We will work on reintroducing the optimizations but that is a multi-week project that won't begin immediately.

The patch is in our release process, building and verifying. It should start rolling it out in a few hours.

@tparker

This comment has been minimized.

Copy link

@tparker tparker commented Apr 26, 2019

Sorry to say we hit a snag in the release process. We are working through it and will have a release ready later this afternoon. But releasing on a Friday afternoon is...dicey. We will wait until Monday to update the servers.

Good news is that the fix bypasses all the finicky calculations that have caused this problem to recur, and we have verified it in a production-like environment.

@tparker

This comment has been minimized.

Copy link

@tparker tparker commented Apr 30, 2019

This was fully deployed by yesterday afternoon, PDT.

@jayconrod

This comment has been minimized.

Copy link
Contributor

@jayconrod jayconrod commented Apr 30, 2019

Thanks @tparker and @draftcode for working on this. I'll close this issue now.

If anyone is still seeing this, please let us know.

@jayconrod jayconrod closed this Apr 30, 2019
@anjmao

This comment has been minimized.

Copy link

@anjmao anjmao commented May 5, 2019

@jayconrod Hi, I still see this issue:

go get -y github.com/jinzhu/gorm

I get error:

go: golang.org/x/net@v0.0.0-20190125091013-d26f9f9a57f3: git fetch -f https://go.googlesource.com/net refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /Users/anjmao/go/pkg/mod/cache/vcs/4a22365141bc4eea5d5ac4a1395e653f2669485db75ef119e7bbec8e19b12a21: exit status 1:
	error: Could not read 65e2d4e15006aab9813ff8769e768bbf4bb667a0
	fatal: bad object 65e2d4e15006aab9813ff8769e768bbf4bb667a0
	error: https://go.googlesource.com/net did not send all necessary objects
@tparker

This comment has been minimized.

Copy link

@tparker tparker commented May 6, 2019

Does anyone have a full reproduction recipe? I'm unable to reproduce it.

@jayconrod

This comment has been minimized.

Copy link
Contributor

@jayconrod jayconrod commented May 10, 2019

@anjmao I couldn't reproduce this. Are you seeing this consistently? Does this happen with an empty module cache (i.e., export GOPATH=$(mktemp -d))? Anything unusual about your environment (git config, git version, proxy settings)?

If you can give us a sequence of git commands that reproduces this (i.e., from go get -x), that would be very helpful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.