cmd/go: hangs with ssh timeouts for https-hosted repo

[oakad]

gopls 0.9.1/go 1.18.4

This is not a new issue, but it appears to be a regression not seen in quite some time.

When trying to open a workspace folder containing many references to normally "unresolvable" packages, gopls gets totally stuck and unable to function while also spamming the VS Code status bar with the error message "context deadline exceeded".

The deadline is exceeded because "git ls-remote " is unable to complete (for whatever unrelated reason). In quite a few preceding gopls versions it was not a problem (the "bad" packages will get a red squiggle and everything else just works), but 0.9.1 somehow gets very upset.

[findleyr]

Thank you for reporting!

• Can you confirm that this is not a problem in v0.8.4 (v0.9.0 is almost the same as v0.9.1).
• Can you see if it's an issue in v0.9.2-pre.2 (go install golang.org/x/tools/gopls@v0.9.2-pre.2)? A lot of loading logic has changed, and while I don't yet understand the root cause, we should confirm that it's still a problem.

Generally speaking, our initial go list request should fail, and then gopls should become responsive, albeit with package errors. It's possible that our logic to retry the initial workspace load has regressed. I will look through CLs to try to guess where this may have happened, but more details would be helpful. Do you see any other errors in your logs? If you have a reliable reproducer, would you be willing to help bisect?

[oakad]

It is pretty curious, may be the problem is with some other tool, not gopls.

Basically, I've got an auto-generated package of a form "some.url/repo.git/package/autogenerated". Those used to work until recently with both "gopls" and "go mod tidy -e" (the former adds squiggles, the later reports an error).

Now I can see "go mod tidy" being stuck and the following processes extant:

go mod tidy -e -compat 1.18

git ls-remote git+ssh://some.url/repo

ssh -o ControlMaster=no -o BatchMode=yes -o SendEnv=GIT_PROTOCOL some.url git-upload-pack 'repo'

"some.url/repo.git" is something that actually exists, only the package is auto generated. Yet, it has no ssh support in there, only https.

[findleyr]

Does go list ./... get stuck? If it went from returning an error to getting stuck, that would explain the regression. Gopls sets an (IMO extremely generous) timeout of 10m for the initial workspace load. It is hard for us to distinguish between slow network and a hanging process like this.

[oakad]

Upon doing further diagnostics I discovered the situation below (I think "some.url" in my case is hosted on AWS)

ssh -vv some.url
debug2: resolving "some.url" port 22
debug1: Connecting to some.url [addr1] port 22.
debug1: connect to address addr1 port 22: Connection timed out
debug1: Connecting to some.url [addr2] port 22.
debug1: connect to address addr2 port 22: Connection timed out
debug1: Connecting to some.url [addr3] port 22.
ad infinitum, AWS has many addresses in its cloudfront pool

On one hand, now I understand this to not be a problem with go tools, but some funny AWS config, recently implemented.

On the other hand, this may affect any sort of package hosted in a similar AWS setup. I don't have the details for the setup, but I know that the repo is supposed to be served on HTTPS, there's no SSH in there.

So, possibly, go should not wait on ssh to complete before trying https, but either try https first, or try them in parallel.

[findleyr]

Thanks for the details.

CC @bcmills to see if there's anything actionable here for cmd/go.

[oakad]

My present work around for the issue was to edit the global .gitconfig as following:

[url "https://some.url"]
        insteadOf = "ssh://some.url"
        insteadOf = "git+ssh://some.url"

Clearly, this is far from perfect, but it gets gopls going. :-)

[seankhliao]

go shouldn't know anything about ssh, it gives git a repo url with http(s). It sounds like the problem is with your git config, which is the part responsible for rewriting https git urls to use ssh.

[oakad]

My git config, until very recently, was trivially simple. The problem has something to do with this:

go/src/cmd/go/internal/vcs/vcs.go

Line 242 in d922c0a

Scheme: []string{"git", "https", "http", "git+ssh", "ssh"},

Namely, it appears to be trying the protocols from this slice in reverse order.

In general, it's a bad idea to do unsolicited calls on other people ssh ports. To name one example, sshd is often used in combination with fail2ban: the later will add the client IP to the iptables blacklist if too many unsuccessful login attempts. There should be a syntax of some sort, possibly in go.mod, to specify the exact protocol to use with each repo.

[oakad]

I shall also add, that "GIT_ALLOW_PROTOCOL=https" also works for me, owning to #17299. Yet, I do think that ability to specify a specific protocol for each package will be a valuable addition to go tooling. Contemporary internet is not a happy place by any measure, with all the networking malice and security countermeasures going around.

[bcmills]

There should be a syntax of some sort, possibly in go.mod, to specify the exact protocol to use with each repo.

There is: the go-import reply from the initial HTTPs fetch can specify a complete URL, including the exact scheme to use.

It is true that we don't have a way to specify the URL scheme when bypassing the go-import protocol by using an explicit .git extension within the import path, but that also doesn't seem to come up very often. 🤔

[oakad]

Yes, and sometimes it's even supported by various vendors (I think gitlab had added some automatic Go friendly redirects into their system). But in a simple case people may want to simply expose some git repo without bothering too much with additional secure http endpoints.

I do appreciate the fact that Go is very open source friendly language trying to encourage people to host their packages openly in public repos. Alas, this world is less than perfect. :-)