Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: in 1.13, the way extracting module vcs address might be incorrect #34339

Closed
Ronmi opened this issue Sep 17, 2019 · 9 comments
Closed

cmd/go: in 1.13, the way extracting module vcs address might be incorrect #34339

Ronmi opened this issue Sep 17, 2019 · 9 comments

Comments

@Ronmi
Copy link

@Ronmi Ronmi commented Sep 17, 2019

What version of Go are you using (go version)?

$ go version
go version go1.13 linux/amd64

Does this issue reproduce with the latest release?

No, go mod tidy runs smoothly in 1.12/1.11

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ronmi/.cache/go-build"
GOENV="/home/ronmi/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY="*.jen.da"
GONOSUMDB="*.jen.da"
GOOS="linux"
GOPATH="/home/ronmi/go"
GOPRIVATE="*.jen.da"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/ronmi/golang/1.13"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/ronmi/golang/1.13/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/ronmi/jenda/richpay_drivers/firstbank/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you do?

We hosted a vcs server

  1. import "git.jen.da/richpay/richpay/common"
  2. go env -w GOPRIVATE=*.jen.da
  3. go mod tidy

What did you expect to see?

It downloads correct content and writes correct record into mod cache, as in go 1.12/1.11

What did you see instead?

It downloads correct contents into $GOPATH/pkg/mod/git.jen.da/richpay/richpay@v0.0.0-20190916085810-25a948e18f6c, but write incorrect git config in mod cache.

Here's the dump of git config in mod cache ($GOPATH/pkg/mod/cache/vcs/c4ba413bf2fb1e97c0b587f6f7a0f095d2fd852b4bb0216247272f4bb68ff7a6)

[core]
        repositoryformatversion = 0
        filemode = true
        bare = true
[remote "origin"]
        url = http://git.jen.da/richpay/.git
        fetch = +refs/heads/*:refs/remotes/origin/*

As mentioned above, the origin.url is supposed to be http://git.jen.da/richpay/richpay.git

Only repo named same as user is affected, importing something like git.jen.da/richpay/tools works smoothly.

@jayconrod

This comment has been minimized.

Copy link
Contributor

@jayconrod jayconrod commented Sep 17, 2019

cc @bcmills

The domain git.jen.da can't be resolved using public DNS, so it may be difficult to reproduce this.

@Ronmi Could you post the output of the commands below:

curl https://git.jen.da/richpay/richpay/common?go-get=1
curl https://git.jen.da/richpay/richpay?go-get=1
curl https://git.jen.da/richpay?go-get=1

The server should respond with a go-import meta tag that indicates where the origin VCS server is. That should more or less literally end up as the remote URL in the git repository.

@Ronmi

This comment has been minimized.

Copy link
Author

@Ronmi Ronmi commented Sep 17, 2019

Thanks for reply.

We use gitea 1.9.1 as git server

$ curl https://git.jen.da/richpay/richpay/common?go-get=1
<!doctype html>
<html>
	<head>
		<meta name="go-import" content="git.jen.da/richpay/richpay git http://git.jen.da/richpay/richpay.git">
		<meta name="go-source" content="git.jen.da/richpay/richpay _ http://git.jen.da/richpay/richpay/src/branch/master{/dir} http://git.jen.da/richpay/richpay/src/branch/master{/dir}/{file}#L{line}">
	</head>
	<body>
		go get git.jen.da/richpay/richpay
	</body>
</html>
$ curl https://git.jen.da/richpay/richpay?go-get=1
<!doctype html>
<html>
	<head>
		<meta name="go-import" content="git.jen.da/richpay/richpay git http://git.jen.da/richpay/richpay.git">
		<meta name="go-source" content="git.jen.da/richpay/richpay _ http://git.jen.da/richpay/richpay/src/branch/master{/dir} http://git.jen.da/richpay/richpay/src/branch/master{/dir}/{file}#L{line}">
	</head>
	<body>
		go get git.jen.da/richpay/richpay
	</body>
</html>
$ curl https://git.jen.da/richpay?go-get=1
<!doctype html>
<html>
	<head>
		<meta name="go-import" content="git.jen.da/richpay git http://git.jen.da/richpay/.git">
		<meta name="go-source" content="git.jen.da/richpay _ http://git.jen.da/richpay/src/branch/master{/dir} http://git.jen.da/richpay/src/branch/master{/dir}/{file}#L{line}">
	</head>
	<body>
		go get git.jen.da/richpay
	</body>
</html>

go.mod in richpay/richpay (running go mod tidy here doesn't cause problem)

module git.jen.da/richpay/richpay

go 1.12

require (
        github.com/DATA-DOG/go-sqlmock v1.3.3 // indirect
        github.com/ericlagergren/decimal v0.0.0-20190912144844-2c3e3e1ef942 // indirect
        github.com/go-sql-driver/mysql v1.4.1
        github.com/gofrs/uuid v3.2.0+incompatible // indirect
        github.com/kat-co/vala v0.0.0-20170210184112-42e1d8b61f12
        github.com/lib/pq v1.2.0 // indirect
        github.com/pkg/errors v0.8.1
        github.com/raohwork/envexist v0.0.1
        github.com/raohwork/jsonapi v0.1.0
        github.com/sirupsen/logrus v1.4.2
        github.com/spf13/viper v1.4.0
        github.com/volatiletech/inflect v0.0.0-20170731032912-e7201282ae8d // indirect
        github.com/volatiletech/null v8.0.0+incompatible
        github.com/volatiletech/sqlboiler v3.5.0+incompatible
)

go.mod in problematic module (but safe with go 1.12)

module git.jen.da/richpay_drivers/firstbank

go 1.12

result of running go mod tidy -v

get "git.jen.da/richpay/richpay/models": found meta tag get.metaImport{Prefix:"git.jen.da/richpay/richpay", VCS:"git", RepoRoot:"http://git.jen.da/richpay/richpay.git"} at //git.jen.da/richpay/richpay/models?go-get=1
get "git.jen.da/richpay/richpay/models": verifying non-authoritative meta tag
get "git.jen.da/richpay/richpay": found meta tag get.metaImport{Prefix:"git.jen.da/richpay/richpay", VCS:"git", RepoRoot:"http://git.jen.da/richpay/richpay.git"} at //git.jen.da/richpay/richpay?go-get=1
get "git.jen.da/richpay/richpay/dba": found meta tag get.metaImport{Prefix:"git.jen.da/richpay/richpay", VCS:"git", RepoRoot:"http://git.jen.da/richpay/richpay.git"} at //git.jen.da/richpay/richpay/dba?go-get=1
get "git.jen.da/richpay/richpay/dba": verifying non-authoritative meta tag
get "git.jen.da/richpay/richpay/models/payment/driver": found meta tag get.metaImport{Prefix:"git.jen.da/richpay/richpay", VCS:"git", RepoRoot:"http://git.jen.da/richpay/richpay.git"} at //git.jen.da/richpay/richpay/models/payment/driver?go-get=1
get "git.jen.da/richpay/richpay/models/payment/driver": verifying non-authoritative meta tag
get "git.jen.da/richpay": found meta tag get.metaImport{Prefix:"git.jen.da/richpay", VCS:"git", RepoRoot:"http://git.jen.da/richpay/.git"} at //git.jen.da/richpay?go-get=1
get "git.jen.da/richpay/richpay/models/payment": found meta tag get.metaImport{Prefix:"git.jen.da/richpay/richpay", VCS:"git", RepoRoot:"http://git.jen.da/richpay/richpay.git"} at //git.jen.da/richpay/richpay/models/payment?go-get=1
get "git.jen.da/richpay/richpay/models/payment": verifying non-authoritative meta tag
get "git.jen.da/richpay/richpay/common": found meta tag get.metaImport{Prefix:"git.jen.da/richpay/richpay", VCS:"git", RepoRoot:"http://git.jen.da/richpay/richpay.git"} at //git.jen.da/richpay/richpay/common?go-get=1
get "git.jen.da/richpay/richpay/common": verifying non-authoritative meta tag
get "git.jen.da": found meta tag get.metaImport{Prefix:"git.jen.da", VCS:"git", RepoRoot:"http://git.jen.da//.git"} at //git.jen.da/?go-get=1
get "git.jen.da/richpay/richpay/models/payment/orders": found meta tag get.metaImport{Prefix:"git.jen.da/richpay/richpay", VCS:"git", RepoRoot:"http://git.jen.da/richpay/richpay.git"} at //git.jen.da/richpay/richpay/models/payment/orders?go-get=1
get "git.jen.da/richpay/richpay/models/payment/orders": verifying non-authoritative meta tag
get "git.jen.da/richpay/richpay/models/payment/results": found meta tag get.metaImport{Prefix:"git.jen.da/richpay/richpay", VCS:"git", RepoRoot:"http://git.jen.da/richpay/richpay.git"} at //git.jen.da/richpay/richpay/models/payment/results?go-get=1
get "git.jen.da/richpay/richpay/dba/paymentdb": found meta tag get.metaImport{Prefix:"git.jen.da/richpay/richpay", VCS:"git", RepoRoot:"http://git.jen.da/richpay/richpay.git"} at //git.jen.da/richpay/richpay/dba/paymentdb?go-get=1
get "git.jen.da/richpay/richpay/dba/paymentdb": verifying non-authoritative meta tag
get "git.jen.da/richpay/richpay/models/payment/results": verifying non-authoritative meta tag
go: finding git.jen.da/richpay/richpay latest
go: finding git.jen.da/richpay/richpay/models latest
go: finding git.jen.da/richpay/richpay/models/payment latest
go: finding git.jen.da/richpay/richpay/common latest
go: finding git.jen.da/richpay/richpay/dba latest
go: finding git.jen.da/richpay/richpay/models/payment/orders latest
go: finding git.jen.da/richpay/richpay/dba/paymentdb latest
go: finding git.jen.da/richpay/richpay/models/payment/driver latest
go: finding git.jen.da/richpay/richpay/models/payment/results latest
go: finding github.com/volatiletech/inflect latest
git.jen.da/richpay_drivers/firstbank imports
	git.jen.da/richpay/richpay/common: git ls-remote -q http://git.jen.da/richpay/.git in /home/ronmi/go/pkg/mod/cache/vcs/c4ba413bf2fb1e97c0b587f6f7a0f095d2fd852b4bb0216247272f4bb68ff7a6: exit status 128:
	fatal: repository 'http://git.jen.da/richpay/.git/' not found
git.jen.da/richpay_drivers/firstbank/vacct imports
	git.jen.da/richpay/richpay/dba/paymentdb: git ls-remote -q http://git.jen.da/richpay/.git in /home/ronmi/go/pkg/mod/cache/vcs/c4ba413bf2fb1e97c0b587f6f7a0f095d2fd852b4bb0216247272f4bb68ff7a6: exit status 128:
	fatal: repository 'http://git.jen.da/richpay/.git/' not found
git.jen.da/richpay_drivers/firstbank/vacct imports
	git.jen.da/richpay/richpay/models/payment/driver: git ls-remote -q http://git.jen.da/richpay/.git in /home/ronmi/go/pkg/mod/cache/vcs/c4ba413bf2fb1e97c0b587f6f7a0f095d2fd852b4bb0216247272f4bb68ff7a6: exit status 128:
	fatal: repository 'http://git.jen.da/richpay/.git/' not found
git.jen.da/richpay_drivers/firstbank/vacct imports
	git.jen.da/richpay/richpay/models/payment/orders: git ls-remote -q http://git.jen.da/richpay/.git in /home/ronmi/go/pkg/mod/cache/vcs/c4ba413bf2fb1e97c0b587f6f7a0f095d2fd852b4bb0216247272f4bb68ff7a6: exit status 128:
	fatal: repository 'http://git.jen.da/richpay/.git/' not found
git.jen.da/richpay_drivers/firstbank/vacct imports
	git.jen.da/richpay/richpay/models/payment/results: git ls-remote -q http://git.jen.da/richpay/.git in /home/ronmi/go/pkg/mod/cache/vcs/c4ba413bf2fb1e97c0b587f6f7a0f095d2fd852b4bb0216247272f4bb68ff7a6: exit status 128:
	fatal: repository 'http://git.jen.da/richpay/.git/' not found

There's no single line imports git.jen.da/richpay

$ grep -F git.jen.da *.go **/*.go|grep -vF '// import'|awk '{print $2}'|sort|uniq
"git.jen.da/richpay_drivers/firstbank"
"git.jen.da/richpay_drivers/firstbank/lib"
"git.jen.da/richpay_drivers/firstbank/libcheck"
"git.jen.da/richpay/richpay/common"
"git.jen.da/richpay/richpay/dba/paymentdb"
"git.jen.da/richpay/richpay/models/payment/driver"
"git.jen.da/richpay/richpay/models/payment/orders"
"git.jen.da/richpay/richpay/models/payment/results"

I also tried few other ways

  1. Adds require git.jen.da/richpay/richpay latest in go.mod manually, everything works.
  2. Remove the richpay line in working go.mod and re-run go mod tidy causes same problem again.
  3. switch back to go 1.12 at anytime also works.
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 17, 2019

This is likely the same underlying issue as #34094 (see #34094 (comment) in particular).

@Ronmi, please try a gotip build and see whether your issue has been resolved. (I suspect that it has.)

@Ronmi

This comment has been minimized.

Copy link
Author

@Ronmi Ronmi commented Sep 17, 2019

Thanks for the references. I'm pretty sure these are same issues, excepts one small detail:

Our gitea server requires users login before accessing any repo (by enabling REQUIRE_SIGNIN_VIEW). After disabling it, gotip fixes the problem while go 1.13 remains failed.

I tried wget https://git.jen.da/richpay/richpay.git/ and wget https://git.jen.da/richpay/.git/, seems gitea does not issue 401 if auth is required. Instead, it uses 302 to redirect user to login page, which is common strategy in web apps.

I also noticed that gitea 1.9.3 fixed response of ?go-get=1 which can solve this problem as well.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 18, 2019

Our gitea server requires users login before accessing any repo (by enabling REQUIRE_SIGNIN_VIEW). After disabling it, gotip fixes the problem while go 1.13 remains failed.

Our expectation is that users are already authenticated to the server (via credentials stored in a .netrc file, or hopefully eventually using the more general GOAUTH protocol described in #26232 (comment).

So you should not have to enable REQUIRE_SIGNIN_VIEW provided that you have appropriate credentials stored in the .netrc file.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 18, 2019

I tried wget https://git.jen.da/richpay/richpay.git/ […]. Instead, it uses 302 to redirect user to login page, which is common strategy in web apps.

As of CL 194561, there are two opportunities for an HTTPS server to signal that a given module path does not exist:

  1. It may elect not to serve a <meta name="go-import"> tag in the initial go-get=1 response, regardless of the status code.
  2. It may serve a <meta […]> tag pointing to a repository with an HTTPS URL, and then serve a 404 or 410 response for that repository.

Following approach (1), as long as the 302'd page does not include a misleading <meta […]> tag, it shouldn't matter what status code it serves for the https://[…].git URLs.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 18, 2019

After disabling it, gotip fixes the problem while go 1.13 remains failed.

Thanks for confirming. In that case, shall I close this as a duplicate of #34094?

I also noticed that gitea 1.9.3 fixed response of ?go-get=1 which can solve this problem as well.

For reference, that seems to be go-gitea/gitea#8095.

@Ronmi

This comment has been minimized.

Copy link
Author

@Ronmi Ronmi commented Sep 20, 2019

After disabling it, gotip fixes the problem while go 1.13 remains failed.

Thanks for confirming. In that case, shall I close this as a duplicate of #34094?

I'm good with it since the root cause is bug of Gitea. Thanks for your help.

I also noticed that gitea 1.9.3 fixed response of ?go-get=1 which can solve this problem as well.

For reference, that seems to be go-gitea/gitea#8095.

Yes. That's where I found the information.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 20, 2019

Duplicate of #34094

@bcmills bcmills marked this as a duplicate of #34094 Sep 20, 2019
@bcmills bcmills closed this Sep 20, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.