Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go get -insecure fetches over http instead of https #34427

Closed
kyroy opened this issue Sep 20, 2019 · 15 comments

Comments

@kyroy
Copy link
Contributor

commented Sep 20, 2019

What version of Go are you using (go version)?

$ go version
go version go1.13 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build880420340=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ docker run  -ti --rm golang:1.13 bash
$ mkdir test && cd test && go mod init test

$ go get -insecure github.xxxx.xxxxx.corp/myorg/go@v0.0.0-20190903123812-3090d622918c
go: finding github.xxxx.xxxxx.corp v0.0.0-20190903123812-3090d622918c
go: finding github.xxxx.xxxxx.corp/myorg v0.0.0-20190903123812-3090d622918c
go: finding github.xxxx.xxxxx.corp/myorg/go v0.0.0-20190903123812-3090d622918c
go get github.xxxx.xxxxx.corp/myorg/go@v0.0.0-20190903123812-3090d622918c: unrecognized import path "github.xxxx.xxxxx.corp/myorg/go" (http/https fetch: Get http://github.xxxx.xxxxx.corp/myorg/go?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout)

$ go get github.xxxx.xxxxx.corp/myorg/go@v0.0.0-20190903123812-3090d622918c
go: finding github.xxxx.xxxxx.corp/myorg/go v0.0.0-20190903123812-3090d622918c
go: finding github.xxxx.xxxxx.corp v0.0.0-20190903123812-3090d622918c
go: finding github.xxxx.xxxxx.corp/myorg v0.0.0-20190903123812-3090d622918c
go get github.xxxx.xxxxx.corp/myorg/go@v0.0.0-20190903123812-3090d622918c: unrecognized import path "github.xxxx.xxxxx.corp/myorg/go" (https fetch: Get https://github.xxxx.xxxxx.corp/myorg/go?go-get=1: x509: certificate signed by unknown authority)

What did you expect to see?

Fetch the information via https but ignore the invalid certificate

What did you see instead?

Go get used http.

@mvdan

This comment has been minimized.

Copy link
Member

commented Sep 20, 2019

This is clearly documented:

$ go help get
[...]
The -insecure flag permits fetching from repositories and resolving
custom domains using insecure schemes such as HTTP. Use with caution.
[...]

Are you proposing that we change what the flag does? If so, wouldn't that break existing users? Imagine if a repository or custom domain doesn't support HTTPS, only HTTP.

@kyroy

This comment has been minimized.

Copy link
Contributor Author

commented Sep 20, 2019

I see, sorry.

How can I then ignore the certificate? Is there any way?

@mvdan

This comment has been minimized.

Copy link
Member

commented Sep 20, 2019

Duplicate of #13197? If https with disabled TLS certificate checks isn't being tried first, this is probably a regression or bug.

/cc @bcmills @jayconrod

@mvdan

This comment has been minimized.

Copy link
Member

commented Sep 20, 2019

Also, please provide the output of one of the go get lines with -v, to see what http/https urls are being tried.

@kyroy

This comment has been minimized.

Copy link
Contributor Author

commented Sep 20, 2019

-v provides exactly the same output

$ go get -v -insecure github.xxxx.xxxxx.corp/myorg/go@v0.0.0-20190903123812-3090d622918c
go: finding github.xxxx.xxxxx.corp v0.0.0-20190903123812-3090d622918c
go: finding github.xxxx.xxxxx.corp/myorg v0.0.0-20190903123812-3090d622918c
go: finding github.xxxx.xxxxx.corp/myorg/go v0.0.0-20190903123812-3090d622918c
go get github.xxxx.xxxxx.corp/myorg/go@v0.0.0-20190903123812-3090d622918c: unrecognized import path "github.xxxx.xxxxx.corp/myorg/go" (http/https fetch: Get http://github.xxxx.xxxxx.corp/myorg/go?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout)
@tmthrgd

This comment has been minimized.

Copy link
Contributor

commented Sep 20, 2019

@kyroy I believe @mvdan meant the -x flag.

@mvdan

This comment has been minimized.

Copy link
Member

commented Sep 20, 2019

Ah, it looks like -x is the right flag nowadays. -v used to give the URLs being queried, but now it's only -x.

@kyroy

This comment has been minimized.

Copy link
Contributor Author

commented Sep 20, 2019

Without GOPRIVATE

$ go get -x -v -insecure github.xxxx.xxxxx.corp/myorg/go@v0.0.0-20190903123812-3090d622918c
go: finding github.xxxx.xxxxx.corp/myorg/go v0.0.0-20190903123812-3090d622918c
go: finding github.xxxx.xxxxx.corp/myorg v0.0.0-20190903123812-3090d622918c
# get https://proxy.golang.org/github.xxxx.xxxxx.corp/myorg/go/@v/v0.0.0-20190903123812-3090d622918c.info
# get https://proxy.golang.org/github.xxxx.xxxxx.corp/myorg/@v/v0.0.0-20190903123812-3090d622918c.info
go: finding github.xxxx.xxxxx.corp v0.0.0-20190903123812-3090d622918c
# get https://proxy.golang.org/github.xxxx.xxxxx.corp/@v/v0.0.0-20190903123812-3090d622918c.info
# get https://proxy.golang.org/github.xxxx.xxxxx.corp/myorg/go/@v/v0.0.0-20190903123812-3090d622918c.info: 410 Gone (7.790s)
# get https://proxy.golang.org/github.xxxx.xxxxx.corp/@v/v0.0.0-20190903123812-3090d622918c.info: 410 Gone (8.441s)
# get https://proxy.golang.org/github.xxxx.xxxxx.corp/myorg/@v/v0.0.0-20190903123812-3090d622918c.info: 410 Gone (8.443s)
# get https://github.xxxx.xxxxx.corp/?go-get=1
# get https://github.xxxx.xxxxx.corp/myorg?go-get=1
# get https://github.xxxx.xxxxx.corp/myorg/go?go-get=1
# get //github.xxxx.xxxxx.corp/?go-get=1: Get https://github.xxxx.xxxxx.corp/?go-get=1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
# get http://github.xxxx.xxxxx.corp/?go-get=1
# get //github.xxxx.xxxxx.corp/myorg?go-get=1: Get https://github.xxxx.xxxxx.corp/myorg?go-get=1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
# get http://github.xxxx.xxxxx.corp/myorg?go-get=1
# get //github.xxxx.xxxxx.corp/myorg/go?go-get=1: Get https://github.xxxx.xxxxx.corp/myorg/go?go-get=1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
# get http://github.xxxx.xxxxx.corp/myorg/go?go-get=1
# get //github.xxxx.xxxxx.corp/myorg/go?go-get=1: Get http://github.xxxx.xxxxx.corp/myorg/go?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout
# get //github.xxxx.xxxxx.corp/?go-get=1: Get http://github.xxxx.xxxxx.corp/?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout
# get //github.xxxx.xxxxx.corp/myorg?go-get=1: Get http://github.xxxx.xxxxx.corp/myorg?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout
go get github.xxxx.xxxxx.corp/myorg/go@v0.0.0-20190903123812-3090d622918c: unrecognized import path "github.xxxx.xxxxx.corp/myorg/go" (http/https fetch: Get http://github.xxxx.xxxxx.corp/myorg/go?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout)

With

$ export GOPRIVATE=github.xxxx.xxxxx.corp
$ go get -x -v -insecure github.xxxx.xxxxx.corp/myorg/go@v0.0.0-20190903123812-3090d622918c
# get https://github.xxxx.xxxxx.corp/?go-get=1
# get https://github.xxxx.xxxxx.corp/myorg/go?go-get=1
# get https://github.xxxx.xxxxx.corp/myorg?go-get=1
# get //github.xxxx.xxxxx.corp/?go-get=1: Get https://github.xxxx.xxxxx.corp/?go-get=1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
# get //github.xxxx.xxxxx.corp/myorg/go?go-get=1: Get https://github.xxxx.xxxxx.corp/myorg/go?go-get=1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
# get http://github.xxxx.xxxxx.corp/myorg/go?go-get=1
# get //github.xxxx.xxxxx.corp/myorg?go-get=1: Get https://github.xxxx.xxxxx.corp/myorg?go-get=1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
# get http://github.xxxx.xxxxx.corp/myorg?go-get=1
# get http://github.xxxx.xxxxx.corp/?go-get=1
# get //github.xxxx.xxxxx.corp/myorg/go?go-get=1: Get http://github.xxxx.xxxxx.corp/myorg/go?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout
# get //github.xxxx.xxxxx.corp/?go-get=1: Get http://github.xxxx.xxxxx.corp/?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout
# get //github.xxxx.xxxxx.corp/myorg?go-get=1: Get http://github.xxxx.xxxxx.corp/myorg?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout
# get https://github.xxxx.xxxxx.corp/myorg?go-get=1
# get https://github.xxxx.xxxxx.corp/myorg/go?go-get=1
# get https://github.xxxx.xxxxx.corp/?go-get=1
# get //github.xxxx.xxxxx.corp/myorg?go-get=1: Get https://github.xxxx.xxxxx.corp/myorg?go-get=1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
# get http://github.xxxx.xxxxx.corp/myorg?go-get=1
# get //github.xxxx.xxxxx.corp/?go-get=1: Get https://github.xxxx.xxxxx.corp/?go-get=1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
# get http://github.xxxx.xxxxx.corp/?go-get=1
# get //github.xxxx.xxxxx.corp/myorg/go?go-get=1: Get https://github.xxxx.xxxxx.corp/myorg/go?go-get=1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
# get http://github.xxxx.xxxxx.corp/myorg/go?go-get=1
# get //github.xxxx.xxxxx.corp/myorg?go-get=1: Get http://github.xxxx.xxxxx.corp/myorg?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout
# get //github.xxxx.xxxxx.corp/?go-get=1: Get http://github.xxxx.xxxxx.corp/?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout
# get //github.xxxx.xxxxx.corp/myorg/go?go-get=1: Get http://github.xxxx.xxxxx.corp/myorg/go?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout
go get github.xxxx.xxxxx.corp/myorg/go@v0.0.0-20190903123812-3090d622918c: unrecognized import path "github.xxxx.xxxxx.corp/myorg/go" (http/https fetch: Get http://github.xxxx.xxxxx.corp/myorg/go?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout)
$ curl -k https://github.xxxx.xxxxx.corp/?go-get=1
<html><body>You are being <a href="https://github.xxxx.xxxxx.corp/repositories">redirected</a>.</body></html>
@toothrot toothrot added this to the Go1.14 milestone Sep 20, 2019
@bcmills

This comment has been minimized.

Copy link
Member

commented Sep 20, 2019

@kyroy, note that the go command follows redirects (curl -kL not just curl -k).

From the trace you've provided, it appears that the go command with -insecure is correctly trying https URLs before falling back to http. You just happen to get the http URL in the error message because that's the last one it tried.

@bcmills

This comment has been minimized.

Copy link
Member

commented Sep 20, 2019

Please try curl -kL https://github.xxxx.xxxxx.corp/myorg/go?go-get=1 and see where you end up.

@kyroy

This comment has been minimized.

Copy link
Contributor Author

commented Sep 23, 2019

Hi, sorry for the delay and thanks for the investigation help :)

$ curl -kL https://github.xxxx.xxxxx.corp/myorg/go?go-get=1

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">



  <title>Go remote import path metadata</title>
  <meta name="go-import" content="github.xxxx.xxxxx.corp/myorg/go git https://github.xxxx.xxxxx.corp/myorg/go.git">



  <link rel="mask-icon" href="https://github.xxxx.xxxxx.corp/pinned-octocat.svg" color="#000000">
  <link rel="icon" type="image/x-icon" class="js-site-favicon" href="https://github.xxxx.xxxxx.corp/favicon-ent.ico">

<meta name="theme-color" content="#1e2327">




  <link rel="manifest" href="/manifest.json" crossOrigin="use-credentials">

  </head>

  <body>

  <!-- Metadata for Go remote import path -->

  </body>
</html>
@bcmills

This comment has been minimized.

Copy link
Member

commented Sep 24, 2019

@kyroy, that doesn't really explain why the connection would be timing out. Is it possible that the server was overloaded or otherwise had very high latency?

@kyroy

This comment has been minimized.

Copy link
Contributor Author

commented Sep 25, 2019

@bcmills Indeed I found out that there has been a bug on the infrastructure in our availability zone that the DNS server was not configured properly. Hence, there was a >5s latency.

Thanks for helping. I am not sure if the error message can be improved. Otherwise, this can be closed :)

(out of scope of this issue)
Another thing that I have experienced a long time ago was that -insecure is not fully enough. You also need to set GIT_SSL_NO_VERIFY. Is there a reason for this / cant this be included?

$ go get -insecure github.xxxx.xxxxx.corp/myorg/go
go get github.xxxx.xxxxx.corp/myorg/go: git ls-remote -q https://github.xxxx.xxxxx.corp/myorg/go.git in /go/pkg/mod/cache/vcs/1e0d9b889f3416a56ea37502ad1137f6723e61f8260c10aaf3fb8c45d44204fe: exit status 128:
	fatal: unable to access 'https://github.xxxx.xxxxx.corp/myorg/go.git/': server certificate verification failed. CAfile: none CRLfile: none

$ GIT_SSL_NO_VERIFY=1 go get -insecure github.xxxx.xxxxx.corp/myorg/go
go: finding github.xxxx.xxxxx.corp/myorg/go latest
go: downloading github.xxxx.xxxxx.corp/myorg/go v0.0.0-20190903123812-3090d622918c
go: extracting github.xxxx.xxxxx.corp/myorg/go v0.0.0-20190903123812-3090d622918c
@bcmills

This comment has been minimized.

Copy link
Member

commented Sep 26, 2019

(out of scope of this issue)

Please file that as a separate issue so that we don't lose track of it. Thanks.

@bcmills bcmills closed this Sep 26, 2019
@kyroy

This comment has been minimized.

Copy link
Contributor Author

commented Sep 27, 2019

Done. Opened #34568

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.