Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: *Request.URL.Scheme returns an empty string. No alternative way present to get request url scheme. #28940

Closed
AnikHasibul opened this issue Nov 25, 2018 · 16 comments

Comments

@AnikHasibul
Copy link

@AnikHasibul AnikHasibul commented Nov 25, 2018

What version of Go are you using (go version)?

$ go version
go version go1.11 linux/amd64

Does this issue reproduce with the latest release?

Yes!

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN="/home/anix/go/bin"
GOCACHE="/home/anix/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/tmp/blackhole:/home/anix/gopath"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build796132705=/tmp/go-build -gno-record-gcc-switches"

What did you do?

package main

import (
        "fmt"
        "net/http"
)

func main() {
        fmt.Println("started!")
        http.HandleFunc("/", redr)
        http.ListenAndServe(":8080", nil)
}

func redr(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintln(w, "SCHEME:", r.URL.Scheme, "HOST:", r.Host, "PATH", r.URL.Path, )
}

What did you expect to see?

$ curl -i http://localhost:8080/
HTTP/1.1 200 OK
Date: Sun, 25 Nov 2018 07:05:42 GMT
Content-Length: 36
Content-Type: text/plain; charset=utf-8

SCHEME: http  HOST: localhost:8080 PATH /

What did you see instead?

The SCHEME: is empty!

$ curl -i http://localhost:8080/
HTTP/1.1 200 OK
Date: Sun, 25 Nov 2018 07:05:42 GMT
Content-Length: 36
Content-Type: text/plain; charset=utf-8

SCHEME:  HOST: localhost:8080 PATH /

Yes!

I read this part of godoc!

$ go doc http.Request.URL

type Request struct {
    // URL specifies either the URI being requested (for server
    // requests) or the URL to access (for client requests).
    //
    // For server requests the URL is parsed from the URI
    // supplied on the Request-Line as stored in RequestURI. For
    // most requests, fields other than Path and RawQuery will be
    // empty. (See RFC 7230, Section 5.3)
    //
    // For client requests, the URL's Host specifies the server to
    // connect to, while the Request's Host field optionally
    // specifies the Host header value to send in the HTTP
    // request.
    URL *url.URL

    // ... other fields elided ...
}

The most focused part is:

// For server requests the URL is parsed from the URI
// supplied on the Request-Line as stored in RequestURI. For
// most requests, fields other than Path and RawQuery will be
// empty. (See RFC 7230, Section 5.3)

But there should be an alternative way to not get an empty string. As r.URL.Host has an alternative r.Host. But what about r.URL.Scheme?

No way!

@tcolgate

This comment has been minimized.

Copy link
Contributor

@tcolgate tcolgate commented Nov 25, 2018

You won't get the scheme here. It is http by virtue of this being a plain (Non-TLS listener). A https request could not have worked.
If you had an extra TLS listener you need to determine which one got hit to infer the scheme, (You can look at the port used to connect, or just use an extra middleware on the TLS listener.

@proglottis

This comment has been minimized.

Copy link

@proglottis proglottis commented Nov 26, 2018

I think checking this field for nil on http.Request is the closest you will get:

        // TLS allows HTTP servers and other software to record
        // information about the TLS connection on which the request
        // was received. This field is not filled in by ReadRequest.
        // The HTTP server in this package sets the field for
        // TLS-enabled connections before invoking a handler;
        // otherwise it leaves the field nil.
        // This field is ignored by the HTTP client.
        TLS *tls.ConnectionState
@tcolgate

This comment has been minimized.

Copy link
Contributor

@tcolgate tcolgate commented Nov 26, 2018

FWIW, since often what you will be wanting this for is to construct a URL non-relative URL for a client, it is desirable to try and respect the x-forwarded-proto header. The client may have used TLS, even though the server processing the request did not receive it over TLS.
@AnikHasibul I'd suggest closing this, it is not a bug.

@andybons

This comment has been minimized.

Copy link
Member

@andybons andybons commented Nov 26, 2018

Thanks.

Checking for a nil TLS field on an http.Request is the correct way to determine if a request is http. We likely will not provide an alternative method since it's redundant to do so given the current solution.

@andybons andybons closed this Nov 26, 2018
@ChrisSalisbury

This comment has been minimized.

Copy link

@ChrisSalisbury ChrisSalisbury commented Apr 23, 2019

Checking TLS is all well and good if the only schemes in existence were http and https, but the real world is far broader than that. I arrived here looking for the best way to check for a ws or wss scheme for proxying purposes. I'm sure others have use cases for other schemes as well.

@tcolgate

This comment has been minimized.

Copy link
Contributor

@tcolgate tcolgate commented Apr 23, 2019

@ChrisSalisbury the scheme simply doesn't exist as part of the HTTP protocol. It's not in the request at all, there's no way to populate the field. The scheme is used when a client interprets a URL for the client to decide which language to use when talking to the server, there's no point where the client is mandate to tell the server what "language" is being used to talk to to it (there's an assumption that if we are communicating at all, then we must already know what language we are talking).

@rzr

This comment has been minimized.

Copy link

@rzr rzr commented Aug 7, 2019

I am new to golang and I found the URL API very confusing,

IMHO, for developer convenience there should be an API somewhere (could be elsewhere than in http or url) to get the full URL from any http(s) request.

I sorted it out by re-constructing from above mentioned hints.

For the record here is a link to mentioned RFC:

https://tools.ietf.org/html/rfc7230#section-5.3

@nicolasparada

This comment has been minimized.

Copy link

@nicolasparada nicolasparada commented Aug 23, 2019

Checking on the request URL TLS does the trick, thanks. But feels weird 😕
Checking the request URL scheme is where anyone would look at.

@lthibault

This comment has been minimized.

Copy link

@lthibault lthibault commented Sep 24, 2019

The suggestions here focus on disambiguating http:// vs https://. How can we determine if an incoming request has e.g. a ws:// scheme?

@tcolgate

This comment has been minimized.

Copy link
Contributor

@tcolgate tcolgate commented Sep 24, 2019

The fundamental point here is that we simply do not know what the scheme is. Scheme is part of the URI spec. As unpalatable as it is, the http requests never gets the URL the client requested, it only gets the path (and, most of the time, the hostname),
Consider the case of a simple TCP based SSL proxy. The connection comes in as https, ssl is stripped, and the connection is forwarded on to the non-ssl server. Which scheme do you want? The one the client plugged into their browser (we can never get that), "https", because the original server saw a TLS connection, or "http".

@DisposaBoy

This comment has been minimized.

Copy link

@DisposaBoy DisposaBoy commented Sep 24, 2019

@lthibault websockets start off as a normal HTTP(S) GET requests that come with headers that look like:

...
Upgrade: websocket
Connection: ...Upgrade...
...
@lthibault

This comment has been minimized.

Copy link

@lthibault lthibault commented Sep 24, 2019

@DisposaBoy Right, but I was hoping to route requests at a given endpoint (e.g. /foo) towards a websocket handler or a vanilla HTTP handler based on the scheme of the incoming request.

Sounds like this isn't possible, so I'll likely have to use a query parameter such as ws://localhost:9021/foo?ws=1. This is ugly because the ws param is redundant with the ws scheme, but what can you do. ¯\_(ツ)_ /¯

@tcolgate

This comment has been minimized.

Copy link
Contributor

@tcolgate tcolgate commented Sep 24, 2019

@lthibault if you just want to know if it's a websocket request, can't you look for the connection upgrade headers?

@DisposaBoy

This comment has been minimized.

Copy link

@DisposaBoy DisposaBoy commented Sep 24, 2019

@lthibault I'm not sure I understand. If you can detect that it's a WS connection by the presence of a ?ws=1 query I don't see why you can't detect it with the upgrade headers.

@tcolgate

This comment has been minimized.

Copy link
Contributor

@tcolgate tcolgate commented Sep 24, 2019

if you want to check ws: vs wss: then you can do both the check for an existing TLS negotiation (or X-Forwarded-For-Scheme), and the upgrade headers.

@lthibault

This comment has been minimized.

Copy link

@lthibault lthibault commented Sep 24, 2019

@DisposaBoy @tcolgate Of course... how did I not think of that. 🤦‍♂

Many thanks!

@jmickey jmickey mentioned this issue Oct 18, 2019
3 of 3 tasks complete
@valtlfelipe valtlfelipe mentioned this issue Jan 23, 2020
2 of 2 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants
You can’t perform that action at this time.