Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: hangs with ssh timeouts for https-hosted repo #54371

Open
oakad opened this issue Aug 10, 2022 · 11 comments
Open

cmd/go: hangs with ssh timeouts for https-hosted repo #54371

oakad opened this issue Aug 10, 2022 · 11 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Tools This label describes issues relating to any tools in the x/tools repository.
Milestone

Comments

@oakad
Copy link

oakad commented Aug 10, 2022

gopls 0.9.1/go 1.18.4

This is not a new issue, but it appears to be a regression not seen in quite some time.

When trying to open a workspace folder containing many references to normally "unresolvable" packages, gopls gets totally stuck and unable to function while also spamming the VS Code status bar with the error message "context deadline exceeded".

The deadline is exceeded because "git ls-remote " is unable to complete (for whatever unrelated reason). In quite a few preceding gopls versions it was not a problem (the "bad" packages will get a red squiggle and everything else just works), but 0.9.1 somehow gets very upset.

@gopherbot gopherbot added Tools This label describes issues relating to any tools in the x/tools repository. gopls Issues related to the Go language server, gopls. labels Aug 10, 2022
@gopherbot gopherbot added this to the Unreleased milestone Aug 10, 2022
@findleyr
Copy link
Contributor

findleyr commented Aug 10, 2022

Thank you for reporting!

  • Can you confirm that this is not a problem in v0.8.4 (v0.9.0 is almost the same as v0.9.1).
  • Can you see if it's an issue in v0.9.2-pre.2 (go install golang.org/x/tools/gopls@v0.9.2-pre.2)? A lot of loading logic has changed, and while I don't yet understand the root cause, we should confirm that it's still a problem.

Generally speaking, our initial go list request should fail, and then gopls should become responsive, albeit with package errors. It's possible that our logic to retry the initial workspace load has regressed. I will look through CLs to try to guess where this may have happened, but more details would be helpful. Do you see any other errors in your logs? If you have a reliable reproducer, would you be willing to help bisect?

@oakad
Copy link
Author

oakad commented Aug 10, 2022

It is pretty curious, may be the problem is with some other tool, not gopls.

Basically, I've got an auto-generated package of a form "some.url/repo.git/package/autogenerated". Those used to work until recently with both "gopls" and "go mod tidy -e" (the former adds squiggles, the later reports an error).

Now I can see "go mod tidy" being stuck and the following processes extant:

go mod tidy -e -compat 1.18

git ls-remote git+ssh://some.url/repo

ssh -o ControlMaster=no -o BatchMode=yes -o SendEnv=GIT_PROTOCOL some.url git-upload-pack 'repo'

"some.url/repo.git" is something that actually exists, only the package is auto generated. Yet, it has no ssh support in there, only https.

@findleyr
Copy link
Contributor

findleyr commented Aug 10, 2022

Does go list ./... get stuck? If it went from returning an error to getting stuck, that would explain the regression. Gopls sets an (IMO extremely generous) timeout of 10m for the initial workspace load. It is hard for us to distinguish between slow network and a hanging process like this.

@oakad
Copy link
Author

oakad commented Aug 10, 2022

Upon doing further diagnostics I discovered the situation below (I think "some.url" in my case is hosted on AWS)

ssh -vv some.url
debug2: resolving "some.url" port 22
debug1: Connecting to some.url [addr1] port 22.
debug1: connect to address addr1 port 22: Connection timed out
debug1: Connecting to some.url [addr2] port 22.
debug1: connect to address addr2 port 22: Connection timed out
debug1: Connecting to some.url [addr3] port 22.
ad infinitum, AWS has many addresses in its cloudfront pool

On one hand, now I understand this to not be a problem with go tools, but some funny AWS config, recently implemented.

On the other hand, this may affect any sort of package hosted in a similar AWS setup. I don't have the details for the setup, but I know that the repo is supposed to be served on HTTPS, there's no SSH in there.

So, possibly, go should not wait on ssh to complete before trying https, but either try https first, or try them in parallel.

@findleyr
Copy link
Contributor

findleyr commented Aug 10, 2022

Thanks for the details.

CC @bcmills to see if there's anything actionable here for cmd/go.

@findleyr findleyr changed the title x/tools/gopls: Error loading workspace: context deadline exceeded cmd/go: hangs with ssh timeouts for https-hosted repo Aug 10, 2022
@findleyr findleyr removed the gopls Issues related to the Go language server, gopls. label Aug 10, 2022
@oakad
Copy link
Author

oakad commented Aug 10, 2022

My present work around for the issue was to edit the global .gitconfig as following:

[url "https://some.url"]
        insteadOf = "ssh://some.url"
        insteadOf = "git+ssh://some.url"

Clearly, this is far from perfect, but it gets gopls going. :-)

@seankhliao
Copy link
Member

seankhliao commented Aug 10, 2022

go shouldn't know anything about ssh, it gives git a repo url with http(s). It sounds like the problem is with your git config, which is the part responsible for rewriting https git urls to use ssh.

@thanm thanm added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 10, 2022
@oakad
Copy link
Author

oakad commented Aug 11, 2022

My git config, until very recently, was trivially simple. The problem has something to do with this:

Scheme: []string{"git", "https", "http", "git+ssh", "ssh"},

Namely, it appears to be trying the protocols from this slice in reverse order.

In general, it's a bad idea to do unsolicited calls on other people ssh ports. To name one example, sshd is often used in combination with fail2ban: the later will add the client IP to the iptables blacklist if too many unsuccessful login attempts. There should be a syntax of some sort, possibly in go.mod, to specify the exact protocol to use with each repo.

@oakad
Copy link
Author

oakad commented Aug 11, 2022

I shall also add, that "GIT_ALLOW_PROTOCOL=https" also works for me, owning to #17299. Yet, I do think that ability to specify a specific protocol for each package will be a valuable addition to go tooling. Contemporary internet is not a happy place by any measure, with all the networking malice and security countermeasures going around.

@bcmills
Copy link
Member

bcmills commented Aug 22, 2022

There should be a syntax of some sort, possibly in go.mod, to specify the exact protocol to use with each repo.

There is: the go-import reply from the initial HTTPs fetch can specify a complete URL, including the exact scheme to use.

It is true that we don't have a way to specify the URL scheme when bypassing the go-import protocol by using an explicit .git extension within the import path, but that also doesn't seem to come up very often. 🤔

@oakad
Copy link
Author

oakad commented Aug 23, 2022

Yes, and sometimes it's even supported by various vendors (I think gitlab had added some automatic Go friendly redirects into their system). But in a simple case people may want to simply expose some git repo without bothering too much with additional secure http endpoints.

I do appreciate the fact that Go is very open source friendly language trying to encourage people to host their packages openly in public repos. Alas, this world is less than perfect. :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Tools This label describes issues relating to any tools in the x/tools repository.
Projects
None yet
Development

No branches or pull requests

6 participants