New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go module not forwarding credentials from .netrc to gitlab subgroups #29888

Open
tigrato opened this Issue Jan 23, 2019 · 8 comments

Comments

Projects
None yet
5 participants
@tigrato
Copy link

tigrato commented Jan 23, 2019

What version of Go are you using (go version)?

$ go 1.11.4

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/tiago/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/tiago/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11.4/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11.4/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/tiago/go/src/gitlab.adinsertp.net/dai-platform/backend-services/media-cache/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/mw/2d5q259n04s64_ylppwvmr9h0000gn/T/go-build740296391=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Last Gitlab release (11.7) enabled go modules downloads for nested groups. It requires authentication for private groups/projects using .netrc file when requesting ?go-get=1 allowing gitlab to understand if the user has the correct access perms for the requested repo.
In case of valid credentials and perms the user should get a go import tag with the project that owns the requested repo.

My $HOME/.netrc file:
machine gitlab.mydomain.com login myuser password mysecrettocken

So I was trying to use go get -v gitlab.mydomain.com/groupA/subGroupB/project/pkg

What did you expect to see?

I expected that go get was able to send the http credentials in .netrc and get the repo that owns the imported package and be able to download it.

What did you see instead?

go get -v gitlab.mydomain.com/groupA/subGroupB/project/pkg Output
Fetching https://gitlab.mydomain.com/groupA/subgroupB/project/pkg?go-get=1
Parsing meta tags from https://gitlab.mydomain.com/groupA/subgroupB/project/pkg?go-get=1 (status code 200)
get "gitlab.mydomain.com/groupA/subgroupB/project": found meta tag get.metaImport{Prefix:"gitlab.mydomain.com/groupA/subgroupB", VCS:"git", RepoRoot:"https://gitlab.mydomain.com/groupA/subgroupB.git"} at https://gitlab.mydomain.com/groupA/subgroupB/project/pkg?go-get=1
get "gitlab.mydomain.com/groupA/subgroupB/project": verifying non-authoritative meta tag
Fetching https://gitlab.mydomain.com/groupA/subgroupB?go-get=1
Parsing meta tags from https://gitlab.mydomain.com/groupA/subgroupB?go-get=1 (status code 200)
Fetching https://gitlab.mydomain.com/groupA/subgroupB?go-get=1
Parsing meta tags from https://gitlab.mydomain.com/groupA/subgroupB?go-get=1 (status code 200)
get "gitlab.mydomain.com/groupA/subgroupB": found meta tag get.metaImport{Prefix:"gitlab.mydomain.com/groupA/subgroupB", VCS:"git", RepoRoot:"https://gitlab.mydomain.com/groupA/subgroupB.git"} at https://gitlab.mydomain.com/groupA/subgroupB?go-get=1
Fetching https://gitlab.mydomain.com/groupA?go-get=1
Parsing meta tags from https://gitlab.mydomain.com/groupA?go-get=1 (status code 200)
Fetching https://gitlab.mydomain.com?go-get=1
Parsing meta tags from https://gitlab.mydomain.com?go-get=1 (status code 200)
go get gitlab.mydomain.com/groupA/subgroupB/project: git ls-remote -q https://gitlab.mydomain.com/groupA/subgroupB.git in /xxx/go/pkg/mod/cache/vcs/5624e7b98ccb36067492e57ba75205fe3d693c9affee381093003e5268dbcaaf: exit status 128:
	remote: The project you were looking for could not be found.
	fatal: repository 'https://gitlab.mydomain.com/groupA/subgroupB.git/' not found

Which fails because it didn't find the correct repository in gitlab. Although if I run:
curl -n https://gitlab.mydomain.com/groupA/subGroupB/project/pkg?go-get=1`:

<html><head><meta name="go-import" content="gitlab.mydomain.com/groupA/subGroupB/project git https://gitlab.mydomain.com/groupA/subGroupB/project.git" /></head></html>
Which returns the correct repository path.

As far I can see, go get is not forwarding .netrc credentials when using subgroups.

@bcmills bcmills added this to the Go1.13 milestone Jan 23, 2019

@bcmills

This comment has been minimized.

Copy link
Member

bcmills commented Jan 24, 2019

Duplicate of #26232?

@vmatyusGitHub

This comment has been minimized.

Copy link

vmatyusGitHub commented Feb 6, 2019

I encounter the same problem: go version go1.11.5 windows/amd64
Is there a go command flag or environment setting that can enable the connection with .netrc?

In the cmd\go\internal\web2\web.go can be found implementation of sending a GET request with .netrc config. But I think they are not used during go get and go build commands.

@bcmills

This comment has been minimized.

Copy link
Member

bcmills commented Feb 7, 2019

No flag should be necessary. Is it possible that your .netrc file is not located in $HOME/.netrc (%USERPROFILE%/_netrc on Windows)?

@vmatyusGitHub

This comment has been minimized.

Copy link

vmatyusGitHub commented Feb 7, 2019

I tried out on Windows and on Linux. The netrc file was at the proper place in both cases.
With curl -n I could receive the correct git repoRoot.

@nilium

This comment has been minimized.

Copy link

nilium commented Feb 7, 2019

From what I can see, go get does not currently use the cmd/go/internal/web2 package that handles netrc loading -- only go mod download would (which has different response expectations than go get, I think?). So, that may explain why go get -v ... doesn't show it -- it's because it doesn't use netrc. Based on #26232, it seems like whether this will even be supported is still in planning.

@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Feb 8, 2019

Change https://golang.org/cl/161698 mentions this issue: cmd/go/internal/web2: make netrc parsing more robust

@tigrato

This comment has been minimized.

Copy link
Author

tigrato commented Feb 12, 2019

From what I can see, go get does not currently use the cmd/go/internal/web2 package that handles netrc loading -- only go mod download would (which has different response expectations than go get, I think?). So, that may explain why go get -v ... doesn't show it -- it's because it doesn't use netrc. Based on #26232, it seems like whether this will even be supported is still in planning.

In this case it is important to have .netrc support for the go get since we can clone the git repo using ssh/git credentials but we cannot discover the repo that owns the package without it.

No flag should be necessary. Is it possible that your .netrc file is not located in $HOME/.netrc (%USERPROFILE%/_netrc on Windows)?
My .netrc is correctly located in $HOME/.netrc as you can see from the curl command with -n

@tigrato

This comment has been minimized.

Copy link
Author

tigrato commented Feb 16, 2019

Some more information

Meta import tag request (...?go-get=1) is done using function

func metaImportsForPrefix(importPrefix string, mod ModuleMode, security web.SecurityMode) (urlStr string, imports []metaImport, err error) {
setCache := func(res fetchResult) (fetchResult, error) {
fetchCacheMu.Lock()
defer fetchCacheMu.Unlock()
fetchCache[importPrefix] = res
return res, nil
}
resi, _, _ := fetchGroup.Do(importPrefix, func() (resi interface{}, err error) {
fetchCacheMu.Lock()
if res, ok := fetchCache[importPrefix]; ok {
fetchCacheMu.Unlock()
return res, nil
}
fetchCacheMu.Unlock()
urlStr, body, err := web.GetMaybeInsecure(importPrefix, security)
if err != nil {
return setCache(fetchResult{urlStr: urlStr, err: fmt.Errorf("fetch %s: %v", urlStr, err)})
}
imports, err := parseMetaGoImports(body, mod)
if err != nil {
return setCache(fetchResult{urlStr: urlStr, err: fmt.Errorf("parsing %s: %v", urlStr, err)})
}
if len(imports) == 0 {
err = fmt.Errorf("fetch %s: no go-import meta tag", urlStr)
}
return setCache(fetchResult{urlStr: urlStr, imports: imports, err: err})
})
res := resi.(fetchResult)
return res.urlStr, res.imports, res.err
}

Which relies on web.GetMayBeInsecure function
func GetMaybeInsecure(importPath string, security SecurityMode) (urlStr string, body io.ReadCloser, err error) {
fetch := func(scheme string) (urlStr string, res *http.Response, err error) {
u, err := url.Parse(scheme + "://" + importPath)
if err != nil {
return "", nil, err
}
u.RawQuery = "go-get=1"
urlStr = u.String()
if cfg.BuildV {
log.Printf("Fetching %s", urlStr)
}
if security == Insecure && scheme == "https" { // fail earlier
res, err = impatientInsecureHTTPClient.Get(urlStr)
} else {
res, err = httpClient.Get(urlStr)
}
return
}
closeBody := func(res *http.Response) {
if res != nil {
res.Body.Close()
}
}
urlStr, res, err := fetch("https")
if err != nil {
if cfg.BuildV {
log.Printf("https fetch failed: %v", err)
}
if security == Insecure {
closeBody(res)
urlStr, res, err = fetch("http")
}
}
if err != nil {
closeBody(res)
return "", nil, err
}
// Note: accepting a non-200 OK here, so people can serve a
// meta import in their http 404 page.
if cfg.BuildV {
log.Printf("Parsing meta tags from %s (status code %d)", urlStr, res.StatusCode)
}
return urlStr, res.Body, nil
}

This last internally uses the http.DefaultClient and does not have any type of authentication or .netrc parsing as happens in the web2 package.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment