Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: get fails on gitlab subgroups due to go-import meta tags referring to nonexistent repos #34094

Open
umputun opened this issue Sep 5, 2019 · 71 comments
Assignees
Milestone

Comments

@umputun
Copy link

@umputun umputun commented Sep 5, 2019

What version of Go are you using (go version)?

$ go version
go version go1.13 darwin/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/umputun/Library/Caches/go-build"
GOENV="/Users/umputun/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/umputun/go-home"
GOPRIVATE=""
GOPROXY="direct"
GOROOT="/usr/local/Cellar/go/1.13/libexec"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.13/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/umputun/tmp/t/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/fx/rzs1_n8137qfktxcbt_2v8pc0000gp/T/go-build570104617=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

  1. Created a public project on gitlab https://gitlab.com/umputuntests/sub/example with a module gitlab.com/umputuntests/sub/example
  2. Tried to get it, i.e. go get -v gitlab.com/umputuntests/sub/example
  3. go get failed
get "gitlab.com/umputuntests/sub/example": found meta tag get.metaImport{Prefix:"gitlab.com/umputuntests/sub/example", VCS:"git", RepoRoot:"https://gitlab.com/umputuntests/sub/example.git"} at //gitlab.com/umputuntests/sub/example?go-get=1
get "gitlab.com/umputuntests/sub": found meta tag get.metaImport{Prefix:"gitlab.com/umputuntests/sub", VCS:"git", RepoRoot:"https://gitlab.com/umputuntests/sub.git"} at //gitlab.com/umputuntests/sub?go-get=1
go: finding gitlab.com/umputuntests/sub/example v0.0.2
go: downloading gitlab.com/umputuntests/sub/example v0.0.2
go: extracting gitlab.com/umputuntests/sub/example v0.0.2
go get gitlab.com/umputuntests/sub/example: git ls-remote -q https://gitlab.com/umputuntests/sub.git in /Users/umputun/go-home/pkg/mod/cache/vcs/410d993df4daac0579ff6b4402c511af12bd3d00851d05ab6183293d03664961: exit status 128:
	fatal: could not read Username for 'https://gitlab.com': terminal prompts disabled
Confirm the import path was entered correctly.
If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.

What did you expect to see?

with prev versions of go (1.12.x) it works file:

 go get -v gitlab.com/umputuntests/sub/example
Fetching https://gitlab.com/umputuntests/sub/example?go-get=1
Parsing meta tags from https://gitlab.com/umputuntests/sub/example?go-get=1 (status code 200)
get "gitlab.com/umputuntests/sub/example": found meta tag get.metaImport{Prefix:"gitlab.com/umputuntests/sub/example", VCS:"git", RepoRoot:"https://gitlab.com/umputuntests/sub/example.git"} at https://gitlab.com/umputuntests/sub/example?go-get=1
go: finding gitlab.com/umputuntests/sub/example v0.0.2
go: finding github.com/stretchr/testify v1.3.0
go: finding github.com/stretchr/objx v0.1.0
go: finding github.com/davecgh/go-spew v1.1.0
go: finding github.com/pmezard/go-difflib v1.0.0
go: downloading gitlab.com/umputuntests/sub/example v0.0.2
go: extracting gitlab.com/umputuntests/sub/example v0.0.2
gitlab.com/umputuntests/sub/example/strategy
gitlab.com/umputuntests/sub/example

additional notes

I think this is both gitlab problem and go 1.13 change. Gitlab seems to return go-import tag on the parent, which doesn't exist and this seems to confuse "go get", trying to check module's parent.

curl "https://gitlab.com/umputuntests/sub?go-get=1"

<html><head><meta name="go-import" content="gitlab.com/umputuntests/sub git https://gitlab.com/umputuntests/sub.git" /></head></html>

In fact, gitlab returns good-looking go-import on any random path, for example:

curl "https://gitlab.com/random/xyz/something?go-get=1"
<html><head><meta name="go-import" content="gitlab.com/random/xyz git https://gitlab.com/random/xyz.git" /></head></html>
@magodo

This comment was marked as off-topic.

Copy link

@magodo magodo commented Sep 5, 2019

Same problem here...

@kinwyb

This comment was marked as off-topic.

Copy link

@kinwyb kinwyb commented Sep 5, 2019

Same problem here

@leonshaw

This comment has been minimized.

Copy link

@leonshaw leonshaw commented Sep 5, 2019

Go 1.13 tries all module prefixes in parallel, and if another error other than "not-found" happens, the query fails. Of course the root cause is on gitlab side, but I think it's reasonable to ignore all errors if at least one attempt succeeds.
A quick workaround is clearing err if found is not empty in queryPrefixModules() in cmd/go/internal/modload/query.go.

@agnivade agnivade changed the title go get v1.13 fails on gitlab subgroups cmd/go: get fails on gitlab subgroups Sep 5, 2019
@agnivade agnivade added this to the Go1.14 milestone Sep 5, 2019
@agnivade

This comment has been minimized.

Copy link
Contributor

@agnivade agnivade commented Sep 5, 2019

@bcmills @jayconrod

EDIT: As a reminder, please note our https://github.com/golang/go/wiki/noplusone policy. Comments like "Me too" do not add anything new to the discussion.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 5, 2019

I think it's reasonable to ignore all errors if at least one attempt succeeds.

I could see doing that if the provided Git URL itself returned a 404 or 410 status code, but note that gitlab.com today does not:

~$ curl -sI https://gitlab.com/nonexistentuser/xyz.git | grep HTTP
HTTP/1.1 401 Unauthorized
@tangtony

This comment has been minimized.

Copy link

@tangtony tangtony commented Sep 5, 2019

I'm running into issues with GitLab subgroups as well, but I'm getting a different error:

$ go get -u example.com/group/subgroup/my-go-project.git
get "example.com/group/subgroup": found meta tag get.metaImport{Prefix:"example.com/group/subgroup", VCS:"git", RepoRoot:"https://example.com/group/subgroup.git"} at //example.com/group/subgroup?go-get=1
go: finding example.com/group/subgroup/my-go-project.git latest
go get example.com/group/subgroup/my-go-project.git: git ls-remote -q https://example.com/group/subgroup.git in C:\Users\dev\go\pkg\mod\cache\vcs\5af58fdf9c9bd0fd298253dde7a0add3f4e5d6be34206b0b46f6805ee365d4ea: exit status 128:
        remote: The project you were looking for could not be found.
        fatal: repository 'https://example.com/group/subgroup.git/' not found

It almost seems like it's assuming that the second part of the path must be the project. Going back to 1.12 fixes it.

Edit: Nevermind, re-ran it with -v and it's the same issue mentioned above with GitLab returning the parent.

@bcmills bcmills added the modules label Sep 5, 2019
@leonshaw

This comment has been minimized.

Copy link

@leonshaw leonshaw commented Sep 5, 2019

@bcmills I mean if a try on gitlab.com/group/project succeeds, we can ignore errors encountered trying gitlab.com/group and gitlab.com/group/project/submodule, no matter what they are.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 5, 2019

@leonshaw, maybe..? The problem with that is that it's asymmetric.

If I have credentials for module example.com/foo containing subdirectory ./bar/baz, and you have credentials for example.com/foo/bar containing subdirectory ./baz, and we both run go get example.com/foo/bar/baz, then we'll end up with different results for the exact same query.

The solution to that is to treat responses other than 404/410 as hard errors. Then I'll fail to fetch example.com/foo/bar, and you'll fail to fetch example.com/foo, and we'll both (consistently!) know that we don't have enough credentials to produce a consistent result.

(More generally: if servers want to avoid leaking information to unauthenticated users, then it's more robust to deny the existence of every module rather than claim the existence of every module — especially now that #29888 is fixed.)

@umputun

This comment has been minimized.

Copy link
Author

@umputun umputun commented Sep 5, 2019

f I have credentials for module example.com/foo containing subdirectory ./bar/baz, and you have credentials for example.com/foo/bar containing subdirectory ./baz, and we both run go get example.com/foo/bar/baz, then we'll end up with different results for the exact same query.

I don't think this is possible, at least with giltab, as they don't have "directory-level permissions" but repository-level only. Probably the same for gihub.

Even in case, if a user has credentials to every repo on the server, go get will fail to get that non-existing parent, but instead of the authorization failure git ls-remote complains about "Could not read from remote repository."

> go get -v example.com/commons/pkg/multierror
get "example.com/commons/pkg": found meta tag get.metaImport{Prefix:"example.com/commons/pkg", VCS:"git", RepoRoot:"https://example.com/commons/pkg.git"} at //example.com/commons/pkg?go-get=1
get "example.com/commons/pkg/multierror": found meta tag get.metaImport{Prefix:"example.com/commons/pkg/multierror", VCS:"git", RepoRoot:"https://example.com/commons/pkg/multierror.git"} at //example.com/commons/pkg/multierror?go-get=1
go get example.com/commons/pkg/multierror: git ls-remote -q https://example.com/commons/pkg.git in /Users/umputun/go-home/pkg/mod/cache/vcs/882b5ac6f69791979e3abd70af30e871e32fe91173f2e25079ef6a88464ec13e: exit status 128:
	> GitLab: The project you were looking for could not be found.
	fatal: Could not read from remote repository.
@leonshaw

This comment has been minimized.

Copy link

@leonshaw leonshaw commented Sep 5, 2019

If I have credentials for module example.com/foo containing subdirectory ./bar/baz, and you have credentials for example.com/foo/bar containing subdirectory ./baz, and we both run go get example.com/foo/bar/baz, then we'll end up with different results for the exact same query.

Is this something more fundamental? (A module can't be uniquely located by an import path.)

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 5, 2019

at least with [gitlab], as they don't have "directory-level permissions" but repository-level only. Probably the same for [github].

GitLab and GitHub are not the only ways to host a module. Ideally, the semantics of the go command should generalize to self-hosted domains with arbitrary structure.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 5, 2019

Is this something more fundamental? (A module can't be uniquely located by an import path.)

A module has a unique path. The mapping of package paths to modules is not guaranteed to be 1:1, although the go command fails with an error message if any one package within the package import graph is provided by more than one module in the module graph.

@leonshaw

This comment has been minimized.

Copy link

@leonshaw leonshaw commented Sep 5, 2019

A module has a unique path. The mapping of package paths to modules is not guaranteed to be 1:1, although the go command fails with an error message if any one package within the package import graph is provided by more than one module in the module graph.

Yes, this is the problem. We need a way to avoid the collision, if possible. If it's simply treated as error, example.com/foo/bar will be unusable if example.com/foo exists.

@umputun

This comment has been minimized.

Copy link
Author

@umputun umputun commented Sep 5, 2019

GitLab and GitHub are not the only ways to host a module. Ideally, the semantics of the go command should generalize to self-hosted domains with arbitrary structure.

Probably, self-hosted giltab used by a majority of self-hosted git systems. However, this new behavior seems to conflict not with giltab only, but gitea as well and may conflict with any system surprised by parent discovery calls. I appreciate the generalized approach and concerns about asymmetrical behavior in some theoretical cases, but the current problem if very practical one.

@bcmills bcmills self-assigned this Sep 5, 2019
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 5, 2019

I have most of a solution coded up, but there is a bad interaction with proxy fallback that I still need to work out.

Even then, my expectation is that a server that wishes to indicate “there is no repository here” (for a repository explicitly mentioned in a go-import <meta> tag) will serve a 404 or a 410 for the URL. I'm not sure whether gitlab.com does that for authenticated users, but for unauthenticated users they're currently serving 401s instead.

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Sep 10, 2019

Change https://golang.org/cl/194560 mentions this issue: cmd/go/internal/modfetch: report the module path for errors in (*codeRepo).Versions

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Sep 10, 2019

Change https://golang.org/cl/194561 mentions this issue: cmd/go/internal/modfetch/codehost: treat nonexistent repositories as “not found”

gopherbot pushed a commit that referenced this issue Sep 10, 2019
…Repo).Versions

Updates #34094

Change-Id: Ifd10b51c2b4ebe77c4f8f68726e411f54c13b9c9
Reviewed-on: https://go-review.googlesource.com/c/go/+/194560
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 10, 2019

@gopherbot, please backport to 1.13: this is a regression, and prevents previously-working modules from being fetched.

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Sep 10, 2019

Backport issue(s) opened: #34215 (for 1.13).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

gopherbot pushed a commit that referenced this issue Sep 11, 2019
…“not found”

If a go-import directive refers to a nonexistent repository, today we
treat that as an error fetching a module that actually exists.
That makes the HTTP server responsible for determining which
repositories do or do not exist, which may in general depend on
the user's separately-stored credentials, and imposes significant
complexity on such a server, which can otherwise be very simple.

Instead, check the repository URL and/or error message to try to
determine whether the repository exists at all. If the repo does not
exist, treat its absence as a “not found” error — as if the server had
not returned it in the first place.

Updates #34094

Change-Id: I142619ff43b96d0de428cdd0b01cca828c9ba234
Reviewed-on: https://go-review.googlesource.com/c/go/+/194561
Reviewed-by: Jay Conrod <jayconrod@google.com>
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 11, 2019

I've implemented the approach described in #34094 (comment). That should solve the problem if GitLab is correctly serving 404 or 410 responses for nonexistent paths for signed-in users.

However, I don't know whether they're doing that, so please help me confirm.

@umputun (or one of the other folks with this problem): please try a go command built from head and let me know whether your problem is resolved. (The easy way to try a go command built from head is to install golang.org/x/dl/gotip, then run gotip download and use gotip in place of the regular go command.)

@GerryLon

This comment was marked as off-topic.

Copy link

@GerryLon GerryLon commented Oct 11, 2019

Same problems...

@bcmills

This comment was marked as off-topic.

Copy link
Member

@bcmills bcmills commented Oct 11, 2019

@bcmills bcmills added Testing and removed WaitingForInfo labels Oct 11, 2019
@bcmills bcmills modified the milestones: Backlog, Go1.14 Oct 11, 2019
@xavivars

This comment has been minimized.

Copy link

@xavivars xavivars commented Oct 12, 2019

Does this mean that this will get fixed for Go1.14, but will still not work for 1.13?

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Oct 12, 2019

@xavivars, this has already been fixed at head and backported to the 1.13 branch. It will be included in Go 1.13.3.

@yookoala

This comment has been minimized.

Copy link

@yookoala yookoala commented Oct 18, 2019

I've just tried with Go 1.13.3 but I still failed in go get my subgroup repository.

$ go version
go version go1.13.3 linux/amd64

$ go get -v gitlab.com/foo/bar/example
get "gitlab.com/foo/bar": found meta tag get.metaImport{Prefix:"gitlab.com/foo/bar", VCS:"git", RepoRoot:"https://gitlab.com/foo/bar.git"} at //gitlab.com/foo/bar?go-get=1
get "gitlab.com/foo/bar/example": found meta tag get.metaImport{Prefix:"gitlab.com/foo/bar", VCS:"git", RepoRoot:"https://gitlab.com/foo/bar.git"} at //gitlab.com/foo/bar/example?go-get=1
get "gitlab.com/foo/bar/example": verifying non-authoritative meta tag
go get gitlab.com/foo/bar/example: git ls-remote -q https://gitlab.com/foo/bar.git in /home/koala/Workspace/gopath/pkg/mod/cache/vcs/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx: exit status 128:
	The project you were looking for could not be found.
	fatal: Could not read from remote repository.
	
	Please make sure you have the correct access rights
	and the repository exists.

P.S. My repository is a private repository. I don't know if it is a matter or not.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Oct 18, 2019

@yookoala, you probably need to configure a git credential helper, or add a .netrc file containing credentials. (The go command is not intended to prompt for interactive logins.)

See https://git-scm.com/docs/gitcredentials and #29953.

@six-arm

This comment was marked as off-topic.

Copy link

@six-arm six-arm commented Oct 18, 2019

I'm still seeing an issue with 1.13.3 on linux. It's pulling the code, but fails with an error.

go version go1.13.3 linux/amd64

GOPRIVATE=gitlab.my-org.com/* go get gitlab.my-org.com/subgroup/test/go-micro-example
go: finding gitlab.my-org.com/subgroup/test/go-micro-example latest
# runtime
/usr/local/go/src/runtime/stubs_x86.go:10:6: stackcheck redeclared in this block
        previous declaration at /usr/local/go/src/runtime/stubs_amd64x.go:10:6
/usr/local/go/src/runtime/unaligned1.go:11:6: readUnaligned32 redeclared in this block
        previous declaration at /usr/local/go/src/runtime/alg.go:321:40
/usr/local/go/src/runtime/unaligned1.go:15:6: readUnaligned64 redeclared in this block
        previous declaration at /usr/local/go/src/runtime/alg.go:329:40

@bcmills

This comment was marked as off-topic.

Copy link
Member

@bcmills bcmills commented Oct 18, 2019

@six-arm, that error message seems unrelated to the issue reported here. Please open a new issue with steps to reproduce.

Given that there is no src/runtime/stubs_x86.go file on release-branch.go1.13, I suspect that there is some problem with your GOROOT setting vs. the go command in use.

@umputun

This comment has been minimized.

Copy link
Author

@umputun umputun commented Oct 19, 2019

go version 1.13.3 fixed the problem. I have tried it against public (gitlab hosted) repo as well as self-hosted giltab. In both cases worked as expected. thx @bcmills and others helped to diagnose and address the problem.

should I close the ticket? This was my first ticket here and I'm not sure what the rule.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Oct 19, 2019

@umputun, thanks for confirming! I'm planning to leave the issue open for adding a more realistic regression test in 1.14 (probably during the freeze).

@GerryLon

This comment has been minimized.

Copy link

@GerryLon GerryLon commented Oct 22, 2019

go version 1.13.3 fixed the problem. I have tried it against public (gitlab hosted) repo as well as self-hosted giltab. In both cases worked as expected. thx @bcmills and others helped to diagnose and address the problem.

should I close the ticket? This was my first ticket here and I'm not sure what the rule.

My go version:

C:\Users\my-gitlab>go version
go version go1.13.3 windows/amd64

simple description:
with private subgroup can not be go get, see below:
my-group are public
my-private-group and it's project sample all are private
my-group and it's project sample all are public
private-demo are private

C:\Users\my-gitlab>go get git2.my-gitlab.net.cn/my-group/my-subgroup/sample
go: finding git2.my-gitlab.net.cn/my-group/my-subgroup/sample latest
go: downloading git2.my-gitlab.net.cn/my-group/my-subgroup/sample v0.0.0-20191022081546-6534fce037e9
go: extracting git2.my-gitlab.net.cn/my-group/my-subgroup/sample v0.0.0-20191022081546-6534fce037e9

C:\Users\my-gitlab>go get git2.my-gitlab.net.cn/my-group/my-private-subgroup/sample
go get git2.my-gitlab.net.cn/my-group/my-private-subgroup/sample: git ls-remote -q origin in D:\workspace\code\go\pkg\mod\cache\vcs\8314af4c2579c0ee8b974f039f8246d179834d34c834680bfe59ce583888c820: exit status 128:
        remote: The project you were looking for could not be found.
        fatal: repository 'https://git2.my-gitlab.net.cn/my-group/my-private-subgroup.git/' not found

C:\Users\my-gitlab>go get git2.my-gitlab.net.cn/my-group/private-demo
go: finding git2.my-gitlab.net.cn/my-group/private-demo latest
go: downloading git2.my-gitlab.net.cn/my-group/private-demo v0.0.0-20191022084509-90eb539586c8
go: extracting git2.my-gitlab.net.cn/my-group/private-demo v0.0.0-20191022084509-90eb539586c8

So, have you tested private group or subgroup on gitlab?

Thanks anyway!

@umputun

This comment has been minimized.

Copy link
Author

@umputun umputun commented Oct 22, 2019

have you tested private group or subgroup on gitlab?

No, I didn't but tried it now with similar setup - commons group public, pkg subgroup public and syncs repo private (internal in gitlab's permissions). I see very similar results:

go version go1.13.3 darwin/amd64

go get -v git.internal.com/commons/pkg/syncs
get "git.internal.com/commons/pkg": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg?go-get=1
get "git.internal.com/commons/pkg/syncs": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg/syncs?go-get=1
get "git.internal.com/commons/pkg/syncs": verifying non-authoritative meta tag
go get git.internal.com/commons/pkg/syncs: git ls-remote -q https://git.internal.com/commons/pkg.git in /Users/umputun/go-home/pkg/mod/cache/vcs/882b5ac6f69791979e3abd70af30e871e32fe91173f2e25079ef6a88464ec13e: exit status 128:
	> GitLab: The project you were looking for could not be found.
	fatal: Could not read from remote repository.

	Please make sure you have the correct access rights
	and the repository exists.

Not sure if important, but to access private repos from go I use insteadOf rewrite:

 [url "git@git.internal.com:"]
     insteadOf = https://git.internal.com/
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Oct 22, 2019

@umputun, thanks for the additional detail.

This line in particular:

get "git.internal.com/commons/pkg/syncs": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg/syncs?go-get=1

seems to indicate a bug or misconfiguration on the server side. If the repo is hosted at git.internal.com/commons/pkg/syncs, then the server should reply with that as the prefix and https://git.internal.com/commons/pkg/syncs.git as the repo root (instead of the git.internal.com/commons/pkg subgroup as shown).

Did that repository work with Go 1.12?

@umputun

This comment has been minimized.

Copy link
Author

@umputun umputun commented Oct 22, 2019

doesn't work with 1.12 either

go version go1.12.12 linux/amd64, from golang:1.12 container:

go get -v git.internal.com/commons/pkg/syncs

Fetching https://git.internal.com/commons/pkg/syncs?go-get=1
Parsing meta tags from https://git.internal.com/commons/pkg/syncs?go-get=1 (status code 200)
get "git.internal.com/commons/pkg/syncs": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at https://git.internal.com/commons/pkg/syncs?go-get=1
get "git.internal.com/commons/pkg/syncs": verifying non-authoritative meta tag
Fetching https://git.internal.com/commons/pkg?go-get=1
Parsing meta tags from https://git.internal.com/commons/pkg?go-get=1 (status code 200)
Fetching https://git.internal.com/commons/pkg?go-get=1
Parsing meta tags from https://git.internal.com/commons/pkg?go-get=1 (status code 200)
get "git.internal.com/commons/pkg": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at https://git.internal.com/commons/pkg?go-get=1
Fetching https://git.internal.com/commons?go-get=1
Parsing meta tags from https://git.internal.com/commons?go-get=1 (status code 200)
Fetching https://git.internal.com?go-get=1
Parsing meta tags from https://git.internal.com?go-get=1 (status code 200)
go get git.internal.com/commons/pkg/syncs: git ls-remote -q origin in /go/pkg/mod/cache/vcs/82b580717104e8b7bf1de06ef2b675a3ca5bf7d044f776fac4629633677f22da: exit status 128:
	> GitLab: The project you were looking for could not be found.
	fatal: Could not read from remote repository.

	Please make sure you have the correct access rights
	and the repository exists.	

However with ~/.netrc everyting seems to work (1.13.3):

go get -v git.internal.com/commons/pkg/syncs
get "git.internal.com/commons/pkg": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg?go-get=1
get "git.internal.com/commons/pkg/syncs": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg/syncs", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg/syncs.git"} at //git.internal.com/commons/pkg/syncs?go-get=1
go: finding git.internal.com/commons/pkg/syncs v1.1.0
go: downloading git.internal.com/commons/pkg/syncs v1.1.0
go: extracting git.internal.com/commons/pkg/syncs v1.1.0
get "git.internal.com/commons/pkg/multierror": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg/multierror", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg/multierror.git"} at //git.internal.com/commons/pkg/multierror?go-get=1
git.internal.com/commons/pkg/multierror
git.internal.com/commons/pkg/syncs	

So, looks like .gitconfig insteadOf doesn't play well with go mods, but .netrc is fine.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Oct 22, 2019

.netrc is what I would recommend anyway, so it sounds like the overall state is improving, at least. (.netrc wasn't working well in 1.12.)

@GerryLon

This comment was marked as off-topic.

Copy link

@GerryLon GerryLon commented Oct 23, 2019

have you tested private group or subgroup on gitlab?

No, I didn't but tried it now with similar setup - commons group public, pkg subgroup public and syncs repo private (internal in gitlab's permissions). I see very similar results:

go version go1.13.3 darwin/amd64

go get -v git.internal.com/commons/pkg/syncs
get "git.internal.com/commons/pkg": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg?go-get=1
get "git.internal.com/commons/pkg/syncs": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg/syncs?go-get=1
get "git.internal.com/commons/pkg/syncs": verifying non-authoritative meta tag
go get git.internal.com/commons/pkg/syncs: git ls-remote -q https://git.internal.com/commons/pkg.git in /Users/umputun/go-home/pkg/mod/cache/vcs/882b5ac6f69791979e3abd70af30e871e32fe91173f2e25079ef6a88464ec13e: exit status 128:
	> GitLab: The project you were looking for could not be found.
	fatal: Could not read from remote repository.

	Please make sure you have the correct access rights
	and the repository exists.

Not sure if important, but to access private repos from go I use insteadOf rewrite:

 [url "git@git.internal.com:"]
     insteadOf = https://git.internal.com/

it can not work as expected.

@sha256x

This comment was marked as off-topic.

Copy link

@sha256x sha256x commented Oct 26, 2019

I'm still facing this issue with go version go1.13.3 darwin/amd64

@bcmills

This comment was marked as off-topic.

Copy link
Member

@bcmills bcmills commented Oct 28, 2019

@sha256x, “this issue” as reported by its original author has been verified as fixed. If you're seeing a similar symptom, please open a new issue with details and steps to reproduce it.

@TheDiveO

This comment was marked as off-topic.

Copy link

@TheDiveO TheDiveO commented Nov 14, 2019

@bcmills @sha256x @umputun There is still the issue where with git-based URL rewriting in combination with GitLab groups you cannot successfully go get ... a repo unless you tell go get in the import path where the git repo actually is, such as foo.bar.org/user/group1/group2/project.git -- and then go modules throw up because the canonical path of that module is foo.bar.org/user/group1/group2/project without any .git adornments. Am I supposed to change the canonical import paths of my private Go modules now to carry that .git adornment?

@TheDiveO

This comment was marked as off-topic.

Copy link

@TheDiveO TheDiveO commented Nov 14, 2019

have you tested private group or subgroup on gitlab?

No, I didn't but tried it now with similar setup - commons group public, pkg subgroup public and syncs repo private (internal in gitlab's permissions). I see very similar results:

go version go1.13.3 darwin/amd64

go get -v git.internal.com/commons/pkg/syncs
get "git.internal.com/commons/pkg": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg?go-get=1
get "git.internal.com/commons/pkg/syncs": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg/syncs?go-get=1
get "git.internal.com/commons/pkg/syncs": verifying non-authoritative meta tag
go get git.internal.com/commons/pkg/syncs: git ls-remote -q https://git.internal.com/commons/pkg.git in /Users/umputun/go-home/pkg/mod/cache/vcs/882b5ac6f69791979e3abd70af30e871e32fe91173f2e25079ef6a88464ec13e: exit status 128:
	> GitLab: The project you were looking for could not be found.
	fatal: Could not read from remote repository.

	Please make sure you have the correct access rights
	and the repository exists.

Not sure if important, but to access private repos from go I use insteadOf rewrite:

 [url "git@git.internal.com:"]
     insteadOf = https://git.internal.com/

it can not work as expected.

Can you please explain? (otherwise, I cannot see how your reply might be of use to those who are not enlightened enough, such as me.)

@MarioCarrion

This comment was marked as off-topic.

Copy link

@MarioCarrion MarioCarrion commented Nov 21, 2019

@TheDiveO We had a similar problem, we solved it by using the replace directive. For example:

require (
	private.gitlab.instance/project/team/somename v1.0.0
)

replace (
	private.gitlab.instance/project/team/somename => private.gitlab.instance/project/team/somename.git v1.0.0
)

However please be aware that this solution does not work cleanly when trying to upgrade that said package to non-tagged versions, for example you would need to do something like:

go get private.gitlab.instance/project/team/somename.git@somecommithash 

Which then will fail with a message similar to:

go: finding private.gitlab.instance/project/team somecommithash
go: finding private.gitlab.instance/project/team/somename.git somecommithash
go: downloading private.gitlab.instance/project/team/somename.git vXYZ-somecommithash
go: extracting private.gitlab.instance/project/team/somename.git vXYZ-somecommithash
go get: private.gitlab.instance/project/team/somename.git@vXYZ-somecommithash: parsing go.mod:
	module declares its path as: private.gitlab.instance/project/team/somename
	        but was required as: private.gitlab.instance/project/team/somename.git

We literally have to copy vXYZ-somecommithash, manually edit go.mod and then overwrite that value, changing it to something like

require (
	private.gitlab.instance/project/team/somename v1.0.0
)

replace (
	private.gitlab.instance/project/team/somename => private.gitlab.instance/project/team/somename.git vXYZ-somecommithash
)
@bcmills

This comment was marked as off-topic.

Copy link
Member

@bcmills bcmills commented Dec 5, 2019

@TheDiveO, @MarioCarrion: if you are having issues with GitLab subgroups for reasons unrelated to the root cause originally reported for this issue, please open a new issue with details and steps to reproduce. Thanks.

@bcmills bcmills changed the title cmd/go: get fails on gitlab subgroups cmd/go: get fails on gitlab subgroups due to go-import meta tags referring to nonexistent repos Dec 5, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.