cmd/go: go mod tidy not using .git/config settings

[noel-yap]

Go version

go version go1.21.5 darwin/arm64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/nyap/Library/Caches/go-build'
GOENV='/Users/nyap/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/nyap/go/pkg/mod'
GONOPROXY='gitlab.com/grailbio/teams/*'
GONOSUMDB='gitlab.com/grailbio/teams/*'
GOOS='darwin'
GOPATH='/Users/nyap/go'
GOPRIVATE='gitlab.com/grailbio/teams/*'
GOPROXY='https://artifactory.ti-apps.aws.grail.com/artifactory/api/go/go,https://proxy.golang.org'
GOROOT='/Users/nyap/.gimme/versions/go1.21.5.darwin.arm64'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/Users/nyap/.gimme/versions/go1.21.5.darwin.arm64/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.21.5'
GCCGO='gccgo'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='0'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/w7/l4dcw2ss1l98wqcxsx1pq03h0000gs/T/go-build3824820425=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

In workspace's .git/config:

[url "ssh://git@gitlab.com"]
	insteadOf = https://gitlab.com

Then go mod tidy.

What did you see happen?

go mod tidy emits lots of no secure protocol found for repository errors:

go: downloading gitlab.com/grailbio/teams/bxds/gsl/core.git v0.3.1
go: grail.com/cmd/bio-gsl imports
	gitlab.com/grailbio/teams/bxds/gsl/core.git/bio/comparator/count: no secure protocol found for repository
go: grail.com/cmd/bio-gsl imports
…

What did you expect to see?

No errors.

After moving the url setting from .git/config to ~/.gitconfig, go mod tidy works as expected.

[bcmills]

The fact that the local .git/config is ignored is as designed. The git repo in which the go command downloads your dependency is not the same repo as the one containing your own module.

However, the need for this insteadOf directive in the first place is unexpected. gitlab.com is not one of the providers affected by #26134, so the go command should instead probe the repo using git ls-remote to determine which protocol to use. There may be a real bug there that we can fix so that you won't need the insteadOf directive in the first place.

[seankhliao]

insteadOf might be there for auth reasons?

[bcmills]

Looking at the code, I think the go command is trying to ping the repo using something like git ls-remote ssh://gitlab.com/user/repo.git.

In contrast, by default GitLab requires the specific user git, which would be a URL like ssh://git@gitlab.com/user/repo.git. We could certainly hard-code a user to try for specific common hosts (at least GitHub and GitLab), but I wonder if there is a more general way to detect which SSH user(s) we should try.

[Jorropo]

More a work around than anything, but I have this exact configuration in ~/.gitconfig for auth reasons:

[url "ssh://git@github.com/"]
       insteadOf = https://github.com/
[url "ssh://git@gitlab.com/"]
       insteadOf = https://gitlab.com/

And it works perfectly with GOPRIVATE and GOPROXY=direct (go module invokes git cli which then do ssh).

[bcmills]

@Jorropo, I'm curious: does it still work if you use insteadOf = ssh://github.com/ in that .gitconfig?

(My hypothesis is that it does, and if so I think the go command should just know to do that itself!)

[Jorropo]

@bcmills it does not:

> GOPRIVATE=github.com go get github.com/ipfs/$REPO@latest
go: github.com/ipfs/$REPO@latest: module github.com/ipfs/$REPO: git ls-remote -q origin in /home/hugo/go/pkg/mod/cache/vcs/1572f4aa1ba63bc6815a9b2d64ca8e071b19e19f33e22b6a95cf4bf0efb40471: exit status 128:
	fatal: could not read Username for 'https://github.com': terminal prompts disabled
Confirm the import path was entered correctly.
If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.
> cat ~/.gitconfig
# ...
[url "ssh://git@github.com/"]
       insteadOf = ssh://github.com/
# ...
> nano ~/.gitconfig # put it back to insteadOf = https://github.com/
> GOPRIVATE=github.com go get github.com/ipfs/$REPO@latest # works
go: downloading github.com/ipfs/$REPO v0.2.20

[bcmills]

@Jorropo, thanks, I guess that's because of #26134. What about for the gitlab.com URL?

[gonnafaraway]

Hi! Problem is actual, i tried to solve problem like this, but it doesn't help
echo "machine $CI_SERVER_HOST login gitlab-ci-token password ${CI_JOB_TOKEN}" > ~/.netrc export GOPRIVATE=gitlab.com/* export GOPROXY=direct export GOSUMDB=off export PKG_CONFIG_PATH="/usr/lib/x86_64-linux-gnu/pkgconfig/" apt update && apt install -y libssl-dev openssl git gcc musl-dev pkg-config git config --global http.sslverify false
can you help?
i think, that the best way, is to make new env for compiler like GOSSL=off or like GOSECURE=off

[yangyile1990]

lint-test-job:   # This job also runs in the test stage.
  stage: test    # It can run at the same time as unit-test-job (in parallel).
  script:
    - echo "Linting code..."
    - git config --global url."http://192.168.88.88:30080/".insteadof "https://gitlab.yyle.com/"
    - git config --global url."http://192.168.88.88:31080/".insteadof "http://gitlab.yyle.com/"
    - git config --global --list
    - go env -w GOPRIVATE=gitlab.yyle.com
    - go env -w GOPROXY=https://proxy.golang.org,direct
    - go mod tidy

no secure protocol found for repository

[SOF3]

I reproduced this issue with ~/.gitconfig as well. I have a custom core.sshcommand configured at scope/origin global file:/root/.gitconfig, but Go (on docker run golang:1.23-alpine3.20, with apk add git 2.45.2) ignores the command I configured. I have to do GIT_SSH_COMMAND=$(git config core.sshcommand) go mod download to workaround this.

This seems a bit unexpected. While it is reasonable that $workspace/.git/config is not relevant to the other repo, it does not make sense that ~/.gitconfig and even /etc/gitconfig are also ignored.

/module # go mod download 2>&1 | head -3
go: my.private/repo@v1.2.3: reading my.private/repo/go.mod at revision v1.2.3: git ls-remote -q origin in /go/pkg/mod/cache/vcs/xxxxxx: exit status 128:
        Host key verification failed.
        fatal: Could not read from remote repository.

/module # GIT_SSH_COMMAND=$(git config core.sshcommand) go mod download
(successful)