Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: 'go build' error from 'git ls-remote' when downloading from private repo #33836

Closed
lihaic opened this issue Aug 26, 2019 · 8 comments
Closed

Comments

@lihaic
Copy link

@lihaic lihaic commented Aug 26, 2019

What version of Go are you using (go version)?

$ go version
go version go1.13rc1 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/lhc/.cache/go-build"
GOENV="/home/lhc/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY="*.bmi"
GONOSUMDB="*.bmi"
GOOS="linux"
GOPATH="/home/lhc/go"
GOPRIVATE="*.bmi"
GOPROXY="https://goproxy.io,direct"
GOROOT="/home/lhc/sdk/go1.13rc1"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/lhc/sdk/go1.13rc1/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/lhc/gitlab.bmi/tss/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build750408901=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I set two enviroment variable first.
$ go1.13rc1 env -w GOPROXY=https://goproxy.io,direct
$ go1.13rc1 env -w GOPRIVATE=*gitlab.bmi
Then i tried to build my project.
$ go1.13rc1 build -v
And then it failed, because some packages my project rely on is in a private repo.
The error message is:

# cd .; git ls-remote -- https://gitlab.bmi/aaa/logv
usage: git ls-remote [--heads] [--tags]  [--upload-pack=]
                     [-q | --quiet] [--exit-code] [--get-url] [ [...]]
# cd .; git ls-remote -- git+ssh://gitlab.bmi/aaa/logv
usage: git ls-remote [--heads] [--tags]  [--upload-pack=]
                     [-q | --quiet] [--exit-code] [--get-url] [ [...]]
# cd .; git ls-remote -- ssh://gitlab.bmi/aaa/logv
usage: git ls-remote [--heads] [--tags]  [--upload-pack=]
                     [-q | --quiet] [--exit-code] [--get-url] [ [...]]
go: gitlab.bmi/aaa/logv.git@v1.2.2: reading gitlab.bmi/aaa/logv.git/go.mod at revision v1.2.2: unknown revision v1.2.2

I thought this git ls-remote is trying to get tag info, is this supposed to be git ls-remote --tags or git ls remote

Then i modified the source code /home/lhc/sdk/go1.13rc1/src/cmd/go/internal/get/vcs.go line186
pingCmd: "ls-remote -- {scheme}://{repo}", ---> to pingCmd: "ls-remote {scheme}://{repo}",

And then i tried to run go1.13rc1 build -v, and it succeed.

What did you expect to see?

I hope after i set GOPROXY and GOPRIVATE, i could download package from both public repo and private repo.

What did you see instead?

But i can't run go build correctly.

Question is that is that line wrong?

Thank you for your help.

@aofei

This comment has been minimized.

Copy link
Contributor

@aofei aofei commented Aug 26, 2019

This seems to work as designed. What you need to know is that go isn't smart enough to handle VCS authentications on its own.

See: Why does "go get" use HTTPS when cloning a repository?

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Aug 26, 2019

Thanks for the report. This may be related to https://golang.org/cl/181237.

Two questions to help us reproduce the problem:

  1. What happens if you try running those ls-remote commands individually on the command line (in an empty temporary directory)?
  2. What version of the git command are you using (as reported by git version)?
@bcmills bcmills added this to the Go1.13 milestone Aug 26, 2019
@bcmills bcmills changed the title go1.13rc1: go build can't download private repo package after set GOPROXY and GOPRIVATE cmd/go: 'go build' error from 'git ls-remote' when downloading from private repo Aug 26, 2019
@lihaic

This comment has been minimized.

Copy link
Author

@lihaic lihaic commented Aug 27, 2019

$git version
git version 2.7.4

Try running those git ls-remote command in an empty directory.

$mkdir emptydir; cd emptydir
$git ls-remote -- https://gitlab.bmi/aaa/logv
usage: git ls-remote [--heads] [--tags]  [--upload-pack=]
                     [-q | --quiet] [--exit-code] [--get-url] [ [...]]
$git ls-remote -- git+ssh://gitlab.bmi/aaa/logv
usage: git ls-remote [--heads] [--tags]  [--upload-pack=]
                     [-q | --quiet] [--exit-code] [--get-url] [ [...]]
#git ls-remote -- ssh://gitlab.bmi/aaa/logv
usage: git ls-remote [--heads] [--tags]  [--upload-pack=]
                     [-q | --quiet] [--exit-code] [--get-url] [ [...]]

Running git ls-remote in an empty directory produces the same result.

$git ls-remote https://gitlab.bmi/aaa/logv
46c721ee6bab43c94dd8b3f3dc8a357cb4884627	HEAD
46c721ee6bab43c94dd8b3f3dc8a357cb4884627	refs/heads/master
1dccbdb4d84588a52a147e9f0e98ce222ba7027a	refs/heads/v1
2741972f4356fd428439411eba51e19639fc4df1	refs/tags/v1.0.0
46c721ee6bab43c94dd8b3f3dc8a357cb4884627	refs/tags/v1.1.0
2c05092bd904db3c5d0f3dfa81059bb0808c317e   refs/tags/v1.2.0
1aeae1413f640f643de9d864f09b36c85d5409ca	refs/tags/v1.2.1
1dccbdb4d84588a52a147e9f0e98ce222ba7027a	refs/tags/v1.2.2
$git ls-remote --tags https://gitlab.bmi/aaa/logv
2741972f4356fd428439411eba51e19639fc4df1	refs/tags/v1.0.0
46c721ee6bab43c94dd8b3f3dc8a357cb4884627	refs/tags/v1.1.0
2c05092bd904db3c5d0f3dfa81059bb0808c317e   refs/tags/v1.2.0
1aeae1413f640f643de9d864f09b36c85d5409ca	refs/tags/v1.2.1
1dccbdb4d84588a52a147e9f0e98ce222ba7027a	refs/tags/v1.2.2

But running git ls-remote without the -- or with --tags get all tag info.

By the way, i order to use private repo, i added ssh git config like below.

$ cat ~/.gitconfig
[user]
	name = lhc
	email = lhcgotobed@163.com
[url "ssh://git@gitlab.bmi/"]
	insteadOf = https://gitlab.bmi/

I think this sholud only avoid go tools using https since my private repo don't have https.
@bcmills Appreciate all you help, thanks.

@lihaic

This comment has been minimized.

Copy link
Author

@lihaic lihaic commented Aug 27, 2019

I am currently using ubuntu 16.04, default git version is 2.7.4.

$sudo add-apt-repository -y ppa:git-core/ppa
$sudo apt-get update
$sudo apt-get install git -y
$git version
git version 2.23.0

I did this to upgrade git to v2.23.0, and then git ls-remote -- worked! And with the origin go1.13rc1 binary, go build works fine with my private repo.
So seems like this problem related to git version.

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Aug 27, 2019

Change https://golang.org/cl/191978 mentions this issue: cmd/go/internal/get: remove '--' separator from 'git ls-remote' command

@bcmills bcmills modified the milestones: Go1.13, Go1.14 Aug 27, 2019
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Aug 27, 2019

@gopherbot, please backport to Go 1.13.

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Aug 27, 2019

Backport issue(s) opened: #33880 (for 1.13).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Aug 27, 2019

Change https://golang.org/cl/191973 mentions this issue: [release-branch.go1.13] cmd/go/internal/get: remove '--' separator from 'git ls-remote' command

gopherbot pushed a commit that referenced this issue Aug 27, 2019
…om 'git ls-remote' command

'git ls-remote' started recognizing the '--' separator at some point
after 2.7.4, but git defaults to version 2.7.4 on Ubuntu 16.04 LTS,
which remains supported by Ubuntu until April 2021.

We added '--' tokens to most VCS commands as a defensive measure in
CL 181237, but it isn't strictly necessary here because the 'scheme'
argument to our template is chosen from a predefined list: we can
safely drop it to retain compatibility.

Cherry-picked from CL 191978.

Updates #33836
Fixes #33880

Change-Id: Ibb53366b95f8029b587e0b7646a439330d759ac7
Reviewed-on: https://go-review.googlesource.com/c/go/+/191973
Run-TryBot: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@gopherbot gopherbot closed this in 0a778cf Aug 27, 2019
tomocy added a commit to tomocy/go that referenced this issue Sep 1, 2019
'git ls-remote' started recognizing the '--' separator at some point
after 2.7.4, but git defaults to version 2.7.4 on Ubuntu 16.04 LTS,
which remains supported by Ubuntu until April 2021.

We added '--' tokens to most VCS commands as a defensive measure in
CL 181237, but it isn't strictly necessary here because the 'scheme'
argument to our template is chosen from a predefined list: we can
safely drop it to retain compatibility.

Fixes golang#33836
Updates golang#26746

Change-Id: Ibb53366b95f8029b587e0b7646a439330d759ac7
Reviewed-on: https://go-review.googlesource.com/c/go/+/191978
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
t4n6a1ka added a commit to t4n6a1ka/go that referenced this issue Sep 5, 2019
'git ls-remote' started recognizing the '--' separator at some point
after 2.7.4, but git defaults to version 2.7.4 on Ubuntu 16.04 LTS,
which remains supported by Ubuntu until April 2021.

We added '--' tokens to most VCS commands as a defensive measure in
CL 181237, but it isn't strictly necessary here because the 'scheme'
argument to our template is chosen from a predefined list: we can
safely drop it to retain compatibility.

Fixes golang#33836
Updates golang#26746

Change-Id: Ibb53366b95f8029b587e0b7646a439330d759ac7
Reviewed-on: https://go-review.googlesource.com/c/go/+/191978
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.