New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: get fails to provide sensible error message for private vcs repos #25982

Open
exploser opened this Issue Jun 20, 2018 · 14 comments

Comments

Projects
None yet
7 participants
@exploser

exploser commented Jun 20, 2018

What did you do?

I have a project that imports a private git repository. When setting it up with vgo get or similar commands, the resolving process will stop abruptly, not writing anything to the disk.

The command succeeds if git credentials are properly set up (*_ASKPASS, global config or the repo is already cloned with the right config, etc.).

What did you expect to see?

An error message, or some indication of what went wrong

What did you see instead?

No error message, just an exit code 1.

System details

go version go1.10.3 linux/amd64 vgo:2018-02-20.1
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/exploser/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/exploser/go"
GORACE=""
GOROOT="/usr/lib/go"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build534162883=/tmp/go-build -gno-record-gcc-switches"
GOROOT/bin/go version: go version go1.10.3 linux/amd64
GOROOT/bin/go tool compile -V: compile version go1.10.3
uname -sr: Linux 4.16.14-2-MANJARO
LSB Version:	n/a
Distributor ID:	ManjaroLinux
Description:	Manjaro Linux
Release:	17.1.10
Codename:	Hakoila
/usr/lib/libc.so.6: GNU C Library (GNU libc) stable release version 2.27.
gdb --version: GNU gdb (GDB) 8.1

@gopherbot gopherbot added this to the vgo milestone Jun 20, 2018

@bcmills bcmills added the NeedsFix label Jun 21, 2018

@rsc rsc modified the milestones: vgo, Go1.11 Jul 12, 2018

@rsc rsc added the modules label Jul 12, 2018

@rsc rsc changed the title from x/vgo: import resolution fails silently for private git repos to cmd/go: import resolution fails silently for private git repos Jul 12, 2018

@rsc rsc changed the title from cmd/go: import resolution fails silently for private git repos to cmd/go: get fails silently for private git repos Jul 17, 2018

@vito-c

This comment has been minimized.

vito-c commented Jul 19, 2018

I was playing around with modules and ran into an error issue similar to the one in this ticket. myprivate/repo is a private repo. I have all of my ssh keys setup correctly and I can clone the repo myself.

Here is the error:

go: converting vendor/manifest: stat github.com/myprivate/repo@1234sha: git fetch -f --depth=1 origin 1234sha:refs/dummy in /Users/vito-c/code/go/src/mod/cache/vcs/456sha: exit status 128:
	fatal: could not read Username for 'https://github.com': terminal prompts disabled

Should I open a new issue for this? I believe it was trying to authenticate over https not via ssh because in my vendor/manifest there is https://github.com/mypriavet/repo If there is any workaround you want me to try let me know.

@wayneashleyberry

This comment has been minimized.

Contributor

wayneashleyberry commented Jul 24, 2018

This also failed for me while running 1.11beta2, not sure if it's related.

$ go1.11beta2 build
go: finding github.com/org/repo v6.9.0
go: errors parsing go.mod:
/foo/bar/go.mod:31: invalid module version "v6.9.0": unknown revision v6.9.0

Where github.com/org/repo is a private repo which can get git clone'd just fine.

@wayneashleyberry

This comment has been minimized.

Contributor

wayneashleyberry commented Aug 4, 2018

Just tried initialising a project that uses dep with beta 3.

$ go1.11beta3 mod init
go: creating new go.mod: module github.com/overhq/over-stories-api
go: copying requirements from Gopkg.lock
go: converting Gopkg.lock: stat github.com/overhq/go-lib@v7.1.2: unknown revision v7.1.2
$ go1.11beta3 build main.go
go: finding github.com/gobuffalo/makr v1.1.0
go: finding github.com/joho/godotenv v1.2.0
go: finding github.com/gobuffalo/pop v0.0.0-20180726201638-c0faa0f7edf8
go: finding github.com/googleapis/gax-go v2.0.0+incompatible
go: finding github.com/sergi/go-diff v1.0.0
go: finding github.com/gobuffalo/uuid v2.0.0+incompatible
go: finding github.com/gobuffalo/packr v1.12.0
go: finding github.com/go-chi/chi v3.3.2+incompatible
go: finding github.com/russross/blackfriday v0.0.0-20180428102519-11635eb403ff
go: finding github.com/gobuffalo/fizz v1.0.2
go: finding github.com/spf13/cobra v0.0.3
go: finding github.com/markbates/going v1.0.0
go: finding github.com/francoispqt/gojay v0.0.0-20180723134641-7dc0f34a209c
go: finding github.com/golang/protobuf v1.1.0
go: finding github.com/shurcooL/github_flavored_markdown v0.0.0-20180602233135-8913699a52e3
go: finding github.com/gobuffalo/packr v1.11.1
go: finding github.com/gobuffalo/validate v1.0.0
go: finding github.com/markbates/inflect v1.0.0
go: finding github.com/mitchellh/go-homedir v0.0.0-20180523094522-3864e76763d9
go: finding github.com/gobuffalo/tags v0.0.0-20180702174034-96c7cdfcadcb
go: finding github.com/go-gorp/gorp v0.0.0-20180410155428-6032c66e0f5f
go: finding github.com/sourcegraph/annotate v0.0.0-20160123013949-f4cad6c6324d
go: finding github.com/spf13/pflag v1.0.1
go: finding github.com/go-redis/redis v6.13.2+incompatible
go: finding github.com/microcosm-cc/bluemonday v1.0.0
go: finding github.com/shurcooL/octicon v0.0.0-20180602230221-c42b0e3b24d9
go: finding github.com/mattn/go-isatty v0.0.3
go: finding golang.org/x/sys v0.0.0-20180616030259-6c888cc515d3
go: finding google.golang.org/api v0.0.0-20180726000515-082d5fa4f1f0
go: finding github.com/go-sql-driver/mysql v1.4.0
go: finding go.uber.org/zap v1.9.0
go: finding github.com/markbates/grift v1.0.0
go: finding google.golang.org/genproto v0.0.0-20180726180014-2a72893556e4
go: finding github.com/serenize/snaker v0.0.0-20171204205717-a683aaf2d516
go: finding gopkg.in/check.v1 v1.0.0-20161208181325-20d25e280405
go: finding gopkg.in/square/go-jose.v2 v2.1.7
go: finding golang.org/x/text v0.3.0
go: finding google.golang.org/appengine v1.1.0
go: finding golang.org/x/sys v0.0.0-20180715085529-ac767d655b30
go: finding github.com/shurcooL/highlight_diff v0.0.0-20170515013008-09bb4053de1b
go: finding github.com/pkg/errors v0.8.0
go: finding github.com/mattn/go-sqlite3 v1.9.0
go: finding github.com/mattn/go-colorable v0.0.9
go: finding github.com/shurcooL/sanitized_anchor_name v0.0.0-20170918181015-86672fcb3f95
go: finding github.com/gobuffalo/validate v2.0.0+incompatible
go: finding cloud.google.com/go v0.23.0
go: finding golang.org/x/oauth2 v0.0.0-20180724155351-3d292e4d0cdc
go: finding go.uber.org/multierr v1.1.0
go: finding github.com/inconshreveable/mousetrap v1.0.0
go: finding github.com/markbates/validate v1.0.0
go: finding dmitri.shuralyov.com/text/kebabcase v0.0.0-20180217051803-40e40b42552a
go: finding github.com/gobuffalo/envy v1.6.3
go: finding github.com/satori/go.uuid v1.2.0
go: finding github.com/gobuffalo/plush v0.0.0-20180619220412-2ec029f415ed
go: finding go.uber.org/atomic v1.3.2
go: finding golang.org/x/sys v0.0.0-20180724212812-e072cadbbdc8
go: finding github.com/mattn/anko v0.0.5
go: finding github.com/sourcegraph/syntaxhighlight v0.0.0-20170531221838-bd320f5d308e
go: finding github.com/kelseyhightower/envconfig v1.3.0
go: finding github.com/fatih/color v1.7.0
go: finding github.com/stretchr/testify v1.2.2
go: finding golang.org/x/net v0.0.0-20180611182652-db08ff08e862
go: finding google.golang.org/grpc v1.13.0
go: finding github.com/cockroachdb/cockroach-go v0.0.0-20180212155653-59c0560478b7
go: finding github.com/shurcooL/highlight_go v0.0.0-20170515013102-78fb10f4a5f8
go: finding golang.org/x/net v0.0.0-20180712202826-d0887baf81f4
go: finding github.com/gobuffalo/uuid v0.0.0-20180207211247-3a9fb6c5c481
go: finding golang.org/x/net v0.0.0-20180724234803-3673e40ba225
go: finding github.com/shurcooL/go v0.0.0-20180423040247-9e1955d9fb6e
go: finding github.com/fatih/structs v1.0.0
go: finding github.com/shurcooL/graphql v0.0.0-20180514000029-62c9ce094e75
go: finding github.com/stretchr/objx v0.1.1
go: finding golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f
go: finding github.com/newrelic/go-agent v2.0.0+incompatible
go: finding github.com/gobuffalo/pop v0.0.0-20180710054625-9936c00d4230
go: finding github.com/lib/pq v0.0.0-20180523175426-90697d60dd84
go: finding gopkg.in/yaml.v2 v2.2.1
go: finding gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
go: finding github.com/gobuffalo/plush v0.0.0-20180724100351-726a3d7f08d2
go: finding golang.org/x/crypto v0.0.0-20180723164146-c126467f60eb
go: finding github.com/jmoiron/sqlx v0.0.0-20180614180643-0dae4fefe7c0
go: finding github.com/shurcooL/go-goon v0.0.0-20170922171312-37c2f522c041
go: finding github.com/inconshreveable/mousetrap v0.0.0-20141017200713-76626ae9c91c
go: finding go.opencensus.io v0.14.0
go: finding github.com/microcosm-cc/bluemonday v0.0.0-20180327211928-995366fdf961
go: finding github.com/fatih/structs v0.0.0-20170103081050-a720dfa8df58
go: finding github.com/gobuffalo/validate v0.0.0-20180210182920-42d8db6e06e6
go: finding github.com/gobuffalo/tags v0.0.0-20180320140155-82cd7696c84a
go: downloading github.com/spf13/cobra v0.0.3
go: downloading github.com/kelseyhightower/envconfig v1.3.0
go: downloading github.com/joho/godotenv v1.2.0
go: downloading github.com/go-redis/redis v6.13.2+incompatible
go: downloading github.com/pkg/errors v0.8.0
go: downloading github.com/gobuffalo/pop v0.0.0-20180726201638-c0faa0f7edf8
go: downloading github.com/francoispqt/gojay v0.0.0-20180723134641-7dc0f34a209c
go: downloading github.com/satori/go.uuid v1.2.0
go: downloading google.golang.org/api v0.0.0-20180726000515-082d5fa4f1f0
go: downloading cloud.google.com/go v0.23.0
go: downloading github.com/go-chi/chi v3.3.2+incompatible
go: downloading github.com/gobuffalo/uuid v2.0.0+incompatible
go: downloading github.com/markbates/inflect v1.0.0
go: downloading github.com/gobuffalo/fizz v1.0.2
go: downloading github.com/spf13/pflag v1.0.1
go: downloading github.com/fatih/color v1.7.0
go: downloading github.com/cockroachdb/cockroach-go v0.0.0-20180212155653-59c0560478b7
go: downloading github.com/lib/pq v0.0.0-20180523175426-90697d60dd84
go: downloading github.com/jmoiron/sqlx v0.0.0-20180614180643-0dae4fefe7c0
go: downloading github.com/markbates/going v1.0.0
go: downloading github.com/go-sql-driver/mysql v1.4.0
go: downloading github.com/gobuffalo/packr v1.12.0
go: downloading github.com/gobuffalo/makr v1.1.0
go: downloading github.com/mattn/go-colorable v0.0.9
go: downloading github.com/mattn/go-isatty v0.0.3
go: downloading github.com/gobuffalo/envy v1.6.3
go: downloading github.com/gobuffalo/plush v0.0.0-20180724100351-726a3d7f08d2
go: downloading github.com/go-gorp/gorp v0.0.0-20180410155428-6032c66e0f5f
go: downloading github.com/gobuffalo/validate v2.0.0+incompatible
go: downloading github.com/gobuffalo/tags v0.0.0-20180702174034-96c7cdfcadcb
go: downloading gopkg.in/yaml.v2 v2.2.1
go: downloading github.com/shurcooL/github_flavored_markdown v0.0.0-20180602233135-8913699a52e3
go: downloading github.com/shurcooL/highlight_go v0.0.0-20170515013102-78fb10f4a5f8
go: downloading golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f
go: downloading github.com/sourcegraph/syntaxhighlight v0.0.0-20170531221838-bd320f5d308e
go: downloading github.com/microcosm-cc/bluemonday v1.0.0
go: downloading golang.org/x/net v0.0.0-20180724234803-3673e40ba225
go: downloading github.com/shurcooL/sanitized_anchor_name v0.0.0-20170918181015-86672fcb3f95
go: downloading github.com/sourcegraph/annotate v0.0.0-20160123013949-f4cad6c6324d
go: downloading github.com/serenize/snaker v0.0.0-20171204205717-a683aaf2d516
go: downloading github.com/russross/blackfriday v0.0.0-20180428102519-11635eb403ff
go: downloading github.com/shurcooL/octicon v0.0.0-20180602230221-c42b0e3b24d9
go: downloading github.com/fatih/structs v1.0.0
go: downloading github.com/shurcooL/highlight_diff v0.0.0-20170515013008-09bb4053de1b
go: downloading github.com/sergi/go-diff v1.0.0
go: downloading github.com/googleapis/gax-go v2.0.0+incompatible
go: downloading go.opencensus.io v0.14.0
go: downloading google.golang.org/grpc v1.13.0
go: downloading google.golang.org/genproto v0.0.0-20180726180014-2a72893556e4
go: downloading github.com/golang/protobuf v1.1.0
go: downloading golang.org/x/text v0.3.0
go: downloading golang.org/x/oauth2 v0.0.0-20180724155351-3d292e4d0cdc
go: import "github.com/overhq/over-stories-api/cmd" ->
	import "github.com/overhq/over-stories-api/cmd/app" ->
	import "github.com/overhq/go-lib/pkg/log": cannot find module providing package github.com/overhq/go-lib/pkg/log
go: import "github.com/overhq/over-stories-api/cmd" ->
	import "github.com/overhq/over-stories-api/cmd/app" ->
	import "github.com/overhq/over-stories-api/pkg/app" ->
	import "github.com/overhq/go-lib/pkg/middleware": cannot find module providing package github.com/overhq/go-lib/pkg/middleware
go: import "github.com/overhq/over-stories-api/cmd" ->
	import "github.com/overhq/over-stories-api/cmd/app" ->
	import "github.com/overhq/over-stories-api/pkg/app" ->
	import "github.com/overhq/go-lib/pkg/servingurl": cannot find module providing package github.com/overhq/go-lib/pkg/servingurl
go: import "github.com/overhq/over-stories-api/cmd" ->
	import "github.com/overhq/over-stories-api/cmd/app" ->
	import "github.com/overhq/over-stories-api/pkg/app" ->
	import "github.com/overhq/over-stories-api/pkg/handler" ->
	import "github.com/overhq/go-lib/pkg/datetime": cannot find module providing package github.com/overhq/go-lib/pkg/datetime
go: import "github.com/overhq/over-stories-api/cmd" ->
	import "github.com/overhq/over-stories-api/cmd/app" ->
	import "github.com/overhq/over-stories-api/pkg/app" ->
	import "github.com/overhq/over-stories-api/pkg/handler" ->
	import "github.com/overhq/go-lib/pkg/errors": cannot find module providing package github.com/overhq/go-lib/pkg/errors
go: import "github.com/overhq/over-stories-api/cmd" ->
	import "github.com/overhq/over-stories-api/cmd/app" ->
	import "github.com/overhq/over-stories-api/pkg/app" ->
	import "github.com/overhq/over-stories-api/pkg/cache" ->
	import "github.com/overhq/over-stories-api/pkg/model" ->
	import "github.com/overhq/go-lib/pkg/strconv": cannot find module providing package github.com/overhq/go-lib/pkg/strconv
@myitcv

This comment has been minimized.

Member

myitcv commented Aug 6, 2018

Just to confirm @wayneashleyberry you're expecting that to succeed because a git clone for, say, github.com/overhq/over-stories-api succeeds?

Are you using ssh-based authentication here or some other?

@wayneashleyberry

This comment has been minimized.

Contributor

wayneashleyberry commented Aug 6, 2018

@myitcv hi 👋

you're expecting that to succeed because a git clone for, say, github.com/overhq/over-stories-api succeeds?

absolutely correct

ssh-based authentication here or some other?

ssh auth

cat ~/.ssh/config
Host *
 AddKeysToAgent yes
 UseKeychain yes
 IdentityFile ~/.ssh/id_rsa
@myitcv

This comment has been minimized.

Member

myitcv commented Aug 6, 2018

Hi indeed @wayneashleyberry!

Do you also have an insteadOf config for git? Something like url.git@github.com:.insteadof=https://github.com/?

@wayneashleyberry

This comment has been minimized.

Contributor

wayneashleyberry commented Aug 6, 2018

nope, what I pasted above is the only config I use for github

@wayneashleyberry

This comment has been minimized.

Contributor

wayneashleyberry commented Aug 6, 2018

Just read https://golang.org/doc/faq#git_https and that fixes things for me! Thanks for pointing me in the right direction @myitcv

@myitcv

This comment has been minimized.

Member

myitcv commented Aug 6, 2018

👍 @wayneashleyberry - we could/should also add to that FAQ section details of how to use keychains (on any platform).

@myitcv

This comment has been minimized.

Member

myitcv commented Aug 7, 2018

So just to summarise what I think the action is here, we need any of the go commands that end up doing anything with a VCS remote (via the corresponding tool), e.g. git ls-remote -q https://github.com/overhq/over-stories-api, to print to stderr the output from the tool in case of a non-zero exit code.

go get is not currently silent (I'll update the issue title) it's just that its output is misleading (comment true as of a9dcbab)

So, taking the example of github.com/overhq/over-stories-api, the current behaviour is:

$ go get github.com/overhq/over-stories-api
go get github.com/overhq/over-stories-api: cannot find module providing package github.com/overhq/over-stories-api

We should instead see something like:

$ go get github.com/overhq/over-stories-api
remote: Repository not found.
fatal: repository 'https://github.com/overhq/over-stories-api/' not found

@myitcv myitcv changed the title from cmd/go: get fails silently for private git repos to cmd/go: get fails to provide sensible error message for private vcs repos Aug 7, 2018

@rsc rsc added the release-blocker label Aug 10, 2018

@rsc

This comment has been minimized.

Contributor

rsc commented Aug 17, 2018

@bradfitz tells me that the "silent exit 1" fix fixed private repos that worked with "old go get". We can address even better support in Go 1.12.

@rsc rsc modified the milestones: Go1.11, Go1.12 Aug 17, 2018

@rsc

This comment has been minimized.

Contributor

rsc commented Oct 25, 2018

In non-module mode I see:

$ go get github.com/overhq/over-stories-api
# cd .; git clone https://github.com/overhq/over-stories-api /Users/rsc/src/github.com/overhq/over-stories-api
Cloning into '/Users/rsc/src/github.com/overhq/over-stories-api'...
fatal: could not read Username for 'https://github.com': terminal prompts disabled
package github.com/overhq/over-stories-api: exit status 128
$ 

In module mode I see:

$ go get github.com/overhq/over-stories-api
go get github.com/overhq/over-stories-api: git ls-remote -q origin in /Users/rsc/pkg/mod/cache/vcs/35b63ced1d50599254b07b2a3912fa8f48350519db24e14283e93ce370969e20: exit status 128:
	fatal: could not read Username for 'https://github.com': terminal prompts disabled
$ 

It's possible that we should recognize this failure (maybe 'terminal prompts disabled') and print a pointer to https://golang.org/doc/faq#git_https.

@rsc

This comment has been minimized.

Contributor

rsc commented Nov 20, 2018

This was a release blocker for Go 1.11 when it was a silent exit, and we fixed that. The extra doc pointer does not block Go 1.12

@rsc rsc removed the release-blocker label Nov 20, 2018

@gopherbot

This comment has been minimized.

gopherbot commented Nov 30, 2018

Change https://golang.org/cl/150777 mentions this issue: cmd/go: link to the FAQ for errors that indicate private VCS repos. Updates #25982

gopherbot pushed a commit that referenced this issue Dec 12, 2018

Julie Qiu
cmd/go: link to the FAQ for errors that indicate private VCS repos.
Updates #25982

Change-Id: I5a284e3844e944f9bfae31fa65b242060d4139c7
Reviewed-on: https://go-review.googlesource.com/c/150777
Run-TryBot: Julie Qiu <julie@golang.org>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment