x/sys Internal Server Error when git clone in aws codebuild image

[outsidebedisdoor]

What version of Go are you using (go version)?

$ go version

go version go1.11 linux

Does this issue reproduce with the latest release?

I am not sure. The code is running in an container codebuild image: aws/codebuild/golang:1.11

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
I have no idea, but you can check follow:
https://github.com/aws/aws-codebuild-docker-images/tree/master/ubuntu/golang/1.11

What did you do?

When I run the code locally, it works as I expect. However, If I run it in code build, when it's git cloning, there seems have internal error.

What did you expect to see?

I expect that it git clone correctly. Also, I run this command with sudo locally, it works fine.

What did you see instead?

GIt clone fails.

[ianlancetaylor]

If at all possible, please cut and paste text rather than posting screenshots. Screenshots are much harder to read and use. Thanks.

This doesn't seem to be a problem with go. It seems to be a problem with running git clone in a codebuild environment. This issue tracker is for problems with Go. For more general issues I suggest that you use a forum; see https://golang.org/wiki/Questions.

[af-engineering]

We're seeing this as well on GCP/Kubernetes.

+ GIT_TRACE=1 go get -v golang.org/x/tools/cmd/goimports
--
2 | Fetching https://golang.org/x/tools/cmd/goimports?go-get=1
3 | Parsing meta tags from https://golang.org/x/tools/cmd/goimports?go-get=1 (status code 200)
4 | get "golang.org/x/tools/cmd/goimports": found meta tag get.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/cmd/goimports?go-get=1
5 | get "golang.org/x/tools/cmd/goimports": verifying non-authoritative meta tag
6 | Fetching https://golang.org/x/tools?go-get=1
7 | Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
8 | golang.org/x/tools (download)
9 | # cd .; git clone https://go.googlesource.com/tools /drone/src/golang.org/x/tools
10 | 18:53:23.160534 git.c:371 trace: built-in: git 'clone' 'https://go.googlesource.com/tools' '/drone/src/golang.org/x/tools'
11 | Cloning into '/drone/src/golang.org/x/tools'...
12 | 18:53:23.167117 run-command.c:350 trace: run_command: 'git-remote-https' 'origin' 'https://go.googlesource.com/tools'
13 | fatal: remote error: Internal Server Error
14 | package golang.org/x/tools/cmd/goimports: exit status 128

+ dep ensure
--
12 | grouped write of manifest, lock and vendor: error while writing out vendor tree: failed to write dep tree: failed to export golang.org/x/crypto:
13 | (1) failed to list versions for https://go.googlesource.com/crypto: fatal: remote error: Internal Server Error
14 | : exit status 128

[ryanchapman]

Maybe this is fixed now? I can't reproduce it anymore. When it was happening, this is what was seen:

GCP Travis-CI build machine:

travis@travis-job-b7a8722c-1e94-40de-b9f9-5fee7df6a25b:~/build/company/docker-images$ curl ifconfig.me ; echo
104.154.113.151

travis@travis-job-b7a8722c-1e94-40de-b9f9-5fee7df6a25b:~/build/company/docker-images$ date -u ; GIT_CURL_VERBOSE=1 GIT_TRACE=1 git clone -vv
v https://go.googlesource.com/net /tmp/z ; date -u
Mon Mar 11 19:12:51 UTC 2019
19:12:51.848019 git.c:344               trace: built-in: git 'clone' '-vvv' 'https://go.googlesource.com/net' '/tmp/z'
Cloning into '/tmp/z'...
19:12:51.850708 run-command.c:626       trace: run_command: 'git-remote-https' 'origin' 'https://go.googlesource.com/net'
* Couldn't find host go.googlesource.com in the .netrc file; using defaults
* Hostname was NOT found in DNS cache
*   Trying 209.85.234.82...
* Connected to go.googlesource.com (209.85.234.82) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
*        server certificate verification OK
*        common name: *.googlecode.com (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: C=US,ST=California,L=Mountain View,O=Google LLC,CN=*.googlecode.com
*        start date: Fri, 01 Mar 2019 09:40:09 GMT
*        expire date: Fri, 24 May 2019 09:25:00 GMT
*        issuer: C=US,O=Google Trust Services,CN=Google Internet Authority G3
*        compression: NULL
*        cipher: AES-128-CBC
*        MAC: SHA1
> GET /net/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/2.15.1
Host: go.googlesource.com
Accept: */*
Accept-Encoding: gzip
Accept-Language: en-US, *;q=0.9
Pragma: no-cache

[... waits for a while...]

< HTTP/1.1 200 OK
< Cache-Control: no-cache, max-age=0, must-revalidate
< Content-Length: 63
< Content-Type: application/x-git-upload-pack-advertisement
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Xss-Protection: 1; mode=block
< Date: Mon, 11 Mar 2019 19:13:52 GMT
<
* Connection #0 to host go.googlesource.com left intact
fatal: remote error: Internal Server Error
Mon Mar 11 19:13:51 UTC 2019

[af-engineering]

We are still getting it.

+ go get golang.org/x/tools/cmd/goimports
--
2 | + go get github.com/gobuffalo/buffalo-plugins
3 | # cd .; git clone https://go.googlesource.com/sys /drone/src/golang.org/x/sys
4 | Cloning into '/drone/src/golang.org/x/sys'...
5 | fatal: remote error: Internal Server Error
6 | package golang.org/x/sys/unix: exit status 128

[af-engineering]

Looks like the same issue as #30690

[ryanchapman]

I'm seeing the issue again. @ianlancetaylor I know it probably isn't the Golang team's responsibility to support git hosting; would you have a team we could contact?

A recent reproduction:

rchapman@Air10111:~$ date -u ; GIT_CURL_VERBOSE=1 GIT_TRACE=1 git clone -vvv https://go.googlesource.com/net /tmp/z ; date -u
Mon Mar 11 20:59:07 UTC 2019
14:59:07.954434 git.c:344               trace: built-in: git 'clone' '-vvv' 'https://go.googlesource.com/net' '/tmp/z'
Cloning into '/tmp/z'...
14:59:07.986770 run-command.c:626       trace: run_command: 'git-remote-https' 'origin' 'https://go.googlesource.com/net'
* Couldn't find host go.googlesource.com in the .netrc file; using defaults
*   Trying 74.125.195.82...
* TCP_NODELAY set
* Connected to go.googlesource.com (74.125.195.82) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=*.googlecode.com
*  start date: Mar  1 09:40:09 2019 GMT
*  expire date: May 24 09:25:00 2019 GMT
*  subjectAltName: host "go.googlesource.com" matched cert's "*.googlesource.com"
*  issuer: C=US; O=Google Trust Services; CN=Google Internet Authority G3
*  SSL certificate verify ok.
> GET /net/info/refs?service=git-upload-pack HTTP/1.1
Host: go.googlesource.com
User-Agent: git/2.15.1
Accept: */*
Accept-Encoding: gzip
Pragma: no-cache

< HTTP/1.1 200 OK
< Cache-Control: no-cache, max-age=0, must-revalidate
< Content-Encoding: gzip
< Content-Type: application/x-git-upload-pack-advertisement
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Xss-Protection: 1; mode=block
< Date: Mon, 11 Mar 2019 20:59:08 GMT
< Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39"
< Transfer-Encoding: chunked
<
* Connection #0 to host go.googlesource.com left intact
14:59:08.925982 run-command.c:626       trace: run_command: 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '-v' '-v' '--check-self-contained-and-connected' '--cloning' 'https://go.googlesource.com/net/'
14:59:08.935224 git.c:344               trace: built-in: git 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '-v' '-v' '--check-self-contained-and-connected' '--cloning' 'https://go.googlesource.com/net/'
Server supports multi_ack_detailed
Server supports no-done
Server supports side-band-64k
Server supports allow-tip-sha1-in-want
Server supports allow-reachable-sha1-in-want
Server supports ofs-delta
Server version is JGit/4-google
want d8887717615a059821345a5c23649351b52a1c0b (refs/heads/master)
want 0ed95abb35c445290478a5348a7b38bb154135fd (refs/heads/release-branch.go1.10)
want a4630153038d3cb8c57f83d95200aea356145cf5 (refs/heads/release-branch.go1.11)
want 65e2d4e15006aab9813ff8769e768bbf4bb667a0 (refs/heads/release-branch.go1.12)
want 62685c2d7ca23c807425dca88b11a3e2323dab41 (refs/heads/release-branch.go1.6)
want d4c55e66d8c3a2f3382d264b08e3e3454a66355a (refs/heads/release-branch.go1.7)
want 186fd3fc8194a5e9980a82230d69c1ff7134229f (refs/heads/release-branch.go1.8)
want 44b7c21cbf19450f38b337eb6b6fe4f6496fb5b3 (refs/heads/release-branch.go1.9)
done
POST git-upload-pack (491 bytes)
* Couldn't find host go.googlesource.com in the .netrc file; using defaults
* Found bundle for host go.googlesource.com: 0x7fe0bcf12430 [can pipeline]
* Re-using existing connection! (#0) with host go.googlesource.com
* Connected to go.googlesource.com (74.125.195.82) port 443 (#0)
> POST /net/git-upload-pack HTTP/1.1
Host: go.googlesource.com
User-Agent: git/2.15.1
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Length: 491

* upload completely sent off: 491 out of 491 bytes
< HTTP/1.1 200 OK
< Cache-Control: no-cache, max-age=0, must-revalidate
< Content-Length: 30
< Content-Type: application/x-git-upload-pack-result
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Xss-Protection: 1; mode=block
< Date: Mon, 11 Mar 2019 21:00:09 GMT
< Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39"
<
* Connection #0 to host go.googlesource.com left intact
fatal: remote error: Internal Server Error
fatal: The remote end hung up unexpectedly
Mon Mar 11 21:00:09 UTC 2019
rchapman@Air10111:~$

[ianlancetaylor]

Let's consolidate discussion of this problem into #30690.