New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cmd/go: validate module proxy URLs received from go-get=1 queries #32006
Comments
~/go/src$ curl -L dmitri.shuralyov.com/app/changes?go-get=1
<meta name="go-import" content="dmitri.shuralyov.com/app/changes git https://dmitri.shuralyov.com/app/changes">
<meta name="go-import" content="dmitri.shuralyov.com/app/changes mod https://">
<meta name="go-source" content="dmitri.shuralyov.com/app/changes https://dmitri.shuralyov.com/app/changes https://gotools.org/dmitri.shuralyov.com/app/changes https://gotools.org/dmitri.shuralyov.com/app/changes#{file}-L{line}"> Note the second line: it includes a bare |
Assigning to Dmitri to either update his server, or make a convincing argument that we should allow the bare |
|
I've edited the original issue to make it more visible that this affects Go tip only. Go 1.12 and Go 1.11 are not affected. I think there are two problems, one in
Since it was only affecting tip and not currently released stable versions of Go, I was using the aforementioned vanity import path as an opportunity to understand the new Go tip behavior better and learn what the best way to handle such URLs would be. I plan to fix the BackgroundThe module proxy protocol documentation did not specify very formally how the URL of the module proxy was joined with the rest of the path. It just said:
The The advantage of that is that it allowed a vanity import path to serve a module at a URL without repetition. I.e., for a vanity import path However, after spending more time thinking about it and discussing with some people, I think the Unfortunately, that means a module with a vanity import path being served by its own domain will have module proxy URLs that are more verbose due to repeating the host, but that seems to be a favorable trade-off to make. I'm happy to hear if anyone has more thoughts on this. |
Previously, the URL of the module proxy used to serve modules beginning with the path dmitri.shuralyov.com contained an empty host value. This worked successfully in Go 1.11 and Go 1.12, leading to GET requests without stuttering. For example, to fetch a list of versions in the dmitri.shuralyov.com/state module, the request was: GET https://dmitri.shuralyov.com/state/@v/list However, a module proxy URL with an empty host no longer works in Go tip as of https://golang.org/cl/170879, and evidence suggests that is the better behavior to continue to enforce. There are plans to document the requirement of a module proxy URL to have a non-empty host. It's also planned to improve the error reporting when the go command encounters an invalid module proxy URLs, so users will have a better idea of what went wrong. See issue golang/go#32006. To be compatible with Go 1.11, 1.12, and the future Go 1.13, this commit changes the module proxy URL used to serve own modules from "https://" to "https://dmitri.shuralyov.com/api/module". Future module proxy requests will look like: GET https://dmitri.shuralyov.com/api/module/dmitri.shuralyov.com/state/@v/list The module proxy protocol is meant primarily for machines rather than humans, so the longer path that stutters is an acceptable compromise. A benefit of the new proxy URL is that we no longer need to check if a request matches. Instead, all HTTP requests with a "/api/module/" path prefix can be immediately recognized as requests to the module proxy, and handled as such. As part of embracing the machine-focused aspect of these URLs, also remove the optional human-friendly behavior of redirecting when the module path and version are not encoded in the request URL. Instead, serve a 400 Bad Request error. A correct module proxy client will not make the mistake of forgetting to encode those values. Updates #24 Updates golang/go#32006
I've made a change to the module proxy URL used to serve modules beginning with the path $ curl -L 'https://dmitri.shuralyov.com/app/changes?go-get=1'
<meta name="go-import" content="dmitri.shuralyov.com/app/changes git https://dmitri.shuralyov.com/app/changes">
<meta name="go-import" content="dmitri.shuralyov.com/app/changes mod https://dmitri.shuralyov.com/api/module">
<meta name="go-source" content="dmitri.shuralyov.com/app/changes https://dmitri.shuralyov.com/app/changes https://gotools.org/dmitri.shuralyov.com/app/changes https://gotools.org/dmitri.shuralyov.com/app/changes#{file}-L{line}"> As a result, those modules can be fetched successfully with Go 1.11, 1.12, and tip. To make it possible to continue to test this issue and make improvements to the error reporting done by the https://dmitri.shuralyov.com/test/modtest2 Here's the current error message from latest Go tip (commit 1ab063c) when trying to fetch it (directly, without a local GOPROXY): $ gotip version
go version devel +1ab063c Fri May 17 22:32:30 2019 +0000 darwin/amd64
$ export GO111MODULE=on
$ export GOPROXY=direct
$ export GOPATH=$(mktemp -d)
$ cd $(mktemp -d)
$ gotip mod init m
go: creating new go.mod: module m
$ gotip get dmitri.shuralyov.com/test/modtest2
go: finding dmitri.shuralyov.com/test/modtest2 v0.0.0
go: dmitri.shuralyov.com/test/modtest2@v0.0.0: Get https:///dmitri.shuralyov.com/test/modtest2/@v/v0.0.0.info: http: no Host in request URL |
Change https://golang.org/cl/191945 mentions this issue: |
This change adds a test module that is served over a module proxy URL with an empty host (i.e., "https://"). It exists only for testing purposes, since such module proxy URLs are not valid as of Go 1.13, and all normal modules are being served with a module proxy URL that has non-empty host as of commit 17791dc. Updates #24 Updates golang/go#32006
What version of Go are you using (
go version
)?Go tip at CL 170879 or later.
Does this issue reproduce with the latest release?
no
What operating system and processor architecture are you using (
go env
)?go env
OutputWhat did you do?
on travis, when running tests against master, modules are unable to be downloaded. i am unsure if this is because of travis or if it can be replicated elsewhere
What did you expect to see?
successful download of modules
What did you see instead?
specifically, the 3x slashes (/) in the url. removing the slash resolves correctly
The text was updated successfully, but these errors were encountered: