cmd/go: support fetching git repos with sha256 hashes

[johnnybubonic]

Go version

go version go1.22.4 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/[REDACTED USER]/.cache/go-build'
GOENV='/home/[REDACTED USER]/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/opt/dev/go/pkg/mod'
GONOPROXY='[REDACTED REPO]/[REDACTED NAME],[REDACTED REPO]/[REDACTED ORG]/*,github.com/[REDACTED ORG2]/*'
GONOSUMDB='[REDACTED REPO]/[REDACTED NAME],[REDACTED REPO]/[REDACTED ORG]/*,github.com/[REDACTED ORG2]/*'
GOOS='linux'
GOPATH='/opt/dev/go'
GOPRIVATE='[REDACTED REPO]/[REDACTED NAME],[REDACTED REPO]/[REDACTED ORG]/*,github.com/[REDACTED ORG2]/*'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/lib/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/lib/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22.4'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1827441876=/tmp/go-build -gno-record-gcc-switches'

What did you do?

Let me preface this that this is likely (also?) an issue with the Go mod proxy, and my apologies if there's already an open issue for this but I looked and didn't see one (though one can imagine searching for "git" "sha256" in the Issues is going to return a lot of unrelated issues regardless for this project).

It is, however, unfortunately painfully apparent that SHA256 commit hashes do not seem to be supported:

(NOTE: Please see this comment; the module shown below has been recreated with SHA1 hashes and a new repository/module has been created to test with.)

$ go mod download r00t2.io/wireproto@v1.0.0
go: r00t2.io/wireproto@v1.0.0: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /opt/dev/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225: exit status 128:
	fatal: Server does not support this repository's object format
	fatal: the remote end hung up unexpectedly
	Gitea: Failed to execute git command

See also here, which provides a bit more useful info.

The same issue is present with 1.23 RC1 as well, of course:

$ go1.23rc1 mod download r00t2.io/wireproto@v1.0.0
go: r00t2.io/wireproto@v1.0.0: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /opt/dev/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225: exit status 128:
	fatal: Server does not support this repository's object format
	fatal: the remote end hung up unexpectedly
	Gitea: Failed to execute git command

The repository itself is fine (I have so-called "vanity" Go module roots/import paths):

$ git clone git@git.r00t2.io:r00t2/go_wireproto && echo '##' ; rm -rf go_wireproto ; git clone https://git.r00t2.io/r00t2/go_wireproto && echo '##' ; cd go_wireproto && git tag -l
Cloning into 'go_wireproto'...
remote: Enumerating objects: 74, done.
remote: Counting objects: 100% (21/21), done.
remote: Compressing objects: 100% (21/21), done.
remote: Total 74 (delta 7), reused 0 (delta 0), pack-reused 53 (from 1)
Receiving objects: 100% (74/74), 61.25 KiB | 1.80 MiB/s, done.
Resolving deltas: 100% (31/31), done.
##
Cloning into 'go_wireproto'...
remote: Enumerating objects: 74, done.
remote: Counting objects: 100% (21/21), done.
remote: Compressing objects: 100% (21/21), done.
remote: Total 74 (delta 7), reused 0 (delta 0), pack-reused 53 (from 1)
Receiving objects: 100% (74/74), 61.25 KiB | 1.53 MiB/s, done.
Resolving deltas: 100% (31/31), done.
##
v1.0.0

The redirect is correct:

$ curl -H "user-agent: Go-http-client/v1" -sL "https://r00t2.io/wireproto?go-get=1" | grep -F 'content='
		<meta name="go-import" content="r00t2.io/wireproto git https://git.r00t2.io/r00t2/go_wireproto">

And it works fine with older repositories still using SHA1 commit hashes.

What did you see happen?

(See above)

What did you expect to see?

SHA256 commit hashes supported (as they've been in upstream git since ~ early 2021-ish?).

[gabyhelp]

Related Issues and Documentation

• cmd/go: go mod download seems now incompatible with git version 1 #51529 (closed)
• cmd/go: go get fails with go modules and with private repositories without a go.mod in Gitea #34075 (closed)
• cmd/go: go get -insecure requires GIT_SSL_NO_VERIFY #34568
• cmd/go: using git over ssh fails to load modules requirements #26894 (closed)
• cmd/go/internal/modconv: TestConvertLegacyConfig/github.com_golang_dep_v0.4.0 broken on Ubuntu 16.04 #26754 (closed)
• cmd/go: git shallow fetches broken at CL 556358 #66147 (closed)
• cmd/go: TestScript/get_update_unknown_protocol test fails [1.15] #43861 (closed)
• affected/package: #56389 (closed)
• modules: go1.11beta2 fails to "find" dependency although regular git succeeds #26568 (closed)
• Problem using go mod download / go get with https git override #63765 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[seankhliao]

The error form the .info endpoint:

not found: r00t2.io/wireproto@v1.0.0: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /tmp/gopath/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225: exit status 128:
	fatal: mismatched algorithms: client sha1; server sha256

is the same as if you used GOPROXY=direct:

$ go get -v -x r00t2.io/wireproto@v1.0.0
# get https://r00t2.io/?go-get=1
# get https://r00t2.io/wireproto?go-get=1
# get https://r00t2.io/?go-get=1: 200 OK (0.775s)
# get https://r00t2.io/wireproto?go-get=1: 200 OK (0.882s)
get "r00t2.io/wireproto": found meta tag vcs.metaImport{Prefix:"r00t2.io/wireproto", VCS:"git", RepoRoot:"https://git.r00t2.io/r00t2/go_wireproto"} at //r00t2.io/wireproto?go-get=1
mkdir -p /home/user/.data/go/pkg/mod/cache/vcs # git3 https://git.r00t2.io/r00t2/go_wireproto
# lock /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225.lock
# /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225 for git3 https://git.r00t2.io/r00t2/go_wireproto
cd /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225; git tag -l
0.004s # cd /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225; git tag -l
cd /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225; git ls-remote -q origin
0.519s # cd /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225; git ls-remote -q origin
cd /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' 15fd8df0bff4b9b0f2c455422e57ef2bc6f8add5c95e045f9b51021874f22440 --
0.004s # cd /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' 15fd8df0bff4b9b0f2c455422e57ef2bc6f8add5c95e045f9b51021874f22440 --
cd /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225; git -c protocol.version=2 fetch -f --depth=1 origin refs/tags/v1.0.0:refs/tags/v1.0.0
0.513s # cd /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225; git -c protocol.version=2 fetch -f --depth=1 origin refs/tags/v1.0.0:refs/tags/v1.0.0
cd /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225; git fetch -f origin 'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*'
0.504s # cd /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225; git fetch -f origin 'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*'
go: r00t2.io/wireproto@v1.0.0: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225: exit status 128:
	fatal: mismatched algorithms: client sha1; server sha256

go fetches the a version using git init + git remote add + fetches,
and the local init will use sha1, resulting in a mismatch with the remote using sha256

https://go.googlesource.com/go/+/e89e880eacb1aceaa14733f44c38e6748d5ffa9c/src/cmd/go/internal/modfetch/codehost/git.go#83

[johnnybubonic]

I needed this library accessible for other projects, so I have deleted the above repository and re-recreated it using SHA1 commit hashes.

In its place, I have turned up a testing repository/module that can be used (and is quite a bit smaller) to aid in testing SHA256 commit hashes with go mod:

https://git.r00t2.io/r00t2/go_sha256repo

The import path is r00t2.io/sha256repo; I have set up an identical import URL for it.

[rsc]

It sounds like we can look at the hash length from ls-remote and then add --objectformat=sha256 to the 'git init' that we do.

[gopherbot]

Change https://go.dev/cl/636475 mentions this issue: cmd/go: add support for git sha256 hashes

[dfinkel]

I put together a quick CL to fix this: https://golang.org/cl/636475

I've marked it WIP because of the 1.24 freeze. (it does have (passing) tests and does work for my a sha256 git repo that I setup in Gitlab a couple weeks ago)

If there's an appetite for a freeze exception, I'll mark it as ready for review.

[dfinkel]

Since we're out of the go 1.24 freeze, I've rebased https://golang.org/cl/636475 and marked it as ready for review.