Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: Question mark eaten from URL if no query parameters specified #13488

Closed
mloskot opened this issue Dec 4, 2015 · 10 comments

Comments

Projects
None yet
5 participants
@mloskot
Copy link

commented Dec 4, 2015

  1. What version of Go are you using (go version)?

    go version go1.5.1 windows/amd64
    
  2. What operating system and processor architecture are you using?

    Windows 8.1 Pro (64-bit)

  3. What did you do?

    There is an (internal) API for which these two URLs identifie two different resources:
    A. http://server.com/api/item.json
    B. http://server.com/api/item.json?

    The second URL is indeed with question mark, but without query parameters.

    I make HTTP GET request using the resource B:

    response, err := http.Get("http://server.com/api/item.json?")
    body, err := ioutil.ReadAll(response.Body)
    response.Body.Close()
    fmt.Printf("%s", body)
    
  4. What did you expect to see?

    I expect to receive representation of the resource B.

  5. What did you see instead?

    I received representation of the resource A.

    So, net/http somewhat eats-or-skips the question mark if there are no query parameters.

    BTW, I'm seeing the same behaviour while using https://github.com/parnurzeal/gorequest package.


I'd like to understand the rationale behind this behaviour better.

Is that because the URL parser interpret the following from RFC 1738, section 3.3, that the "?" (question mark) is not part of the query string, but just separates it?

http://<host>:<port>/<path>?<searchpart>

Does also the grammar in RFC 3986, Appendix A., indicate the "?" is not part of the actual query string?

URI           = scheme ":" hier-part [ "?" query ] [ "#" fragment ]

If so, is my assumption correct that the net/http package decomposes and recomposes URL like http://server.com/item.json? applying the suggested (de|re)composition algorithm, i.e.:

      if defined(query) then
         append "?" to result;
         append query to result;
      endif;

and, rightly, assumes the query component as undefined, hence superfluous?


BTW, I have filed very similar report to the Python Request library (kennethreitz/requests#2912).
I have also tested the requests using number of HTTP packages for different programming languages, here https://github.com/mloskot/http-url-test.
To summary, 4 of 14 packages eat-or-skip the trailing question mark, namely:

@bradfitz

This comment has been minimized.

Copy link
Member

commented Dec 4, 2015

This is the first I've heard of anybody caring about this. That's a pretty delicate API.

As background, we just basically kinda have no place to put this info:
http://play.golang.org/p/4RZwHtWKuV

    u, err := url.Parse("http://example.com/foo")
    fmt.Printf("%#v, %v\n", u, err)

    u, err = url.Parse("http://example.com/foo?")
    fmt.Printf("%#v, %v\n", u, err)

    u, err = url.Parse("http://example.com/foo?a=b")
    fmt.Printf("%#v, %v\n", u, err)

Currently says:

&url.URL{Scheme:"http", Opaque:"", User:(*url.Userinfo)(nil), Host:"example.com", Path:"/foo", RawPath:"", RawQuery:"", Fragment:""}, <nil>
&url.URL{Scheme:"http", Opaque:"", User:(*url.Userinfo)(nil), Host:"example.com", Path:"/foo", RawPath:"", RawQuery:"", Fragment:""}, <nil>
&url.URL{Scheme:"http", Opaque:"", User:(*url.Userinfo)(nil), Host:"example.com", Path:"/foo", RawPath:"", RawQuery:"a=b", Fragment:""}, <nil>

I suppose we could special-case this with RawPath or Opaque, but I'd kinda rather not.

/cc @rsc for opinions.

@mloskot

This comment has been minimized.

Copy link
Author

commented Dec 4, 2015

@bradfitz

That's a pretty delicate API.

I sympathise with your opinion.

I'd kinda rather not.

And, I would understand the rationale.

Via those issue reports, I'm trying to learn what's vendors' take on distinguishing such URLs, as I'm struggling to draw any strict rule from the RFCs.

(Apology for lacking Go fu, I've just kickstarted myself to Go in this mloskot/http-url-test exercise.)

@mloskot

This comment has been minimized.

Copy link
Author

commented Dec 6, 2015

I received some pointers to further details in RFC 3986 (see Is question mark in URL part of query string?). In particular, 6.2.3. Scheme-Based Normalization which somewhat applies to this issue and the "question mark" handling. Quoting:

Normalization should not remove delimiters when their associated component is empty unless licensed to do so by the scheme specification. For example, the URI http://example.com/? cannot be assumed to be equivalent to any of the examples above.

Where "examples above" refers to:

http://example.com
http://example.com/
http://example.com:/
http://example.com:80/


FYI, here is an example of API similar to the one in my issue above:

https://confluence.ucop.edu/display/Curation/ARK

a brief metadata record if you append a single question mark to the ARK
(...)
Adding '??' to the end should return a policy statement.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Dec 28, 2015

As people seem to agree, distinguishing these two is very likely a mistake. You say it is an internal API. I don't believe there are any examples of external APIs that demonstrate this problem, so we're not likely to make changes for a single non-public (and therefore not widely used) server.

What about http://server.com/api/item.json?aegjadrlfgdks= or http://server/api/item.json?1 ? Does the server ignore unexpected variables, or is that yet another page?

It is still possible to create an HTTP request explicitly and then modify its URL to set the Opaque field to "/api/item.json?", if you must make calls to the server this way.

I don't think we should take any steps to help the situation unless there is evidence that it's a common problem for Go users. It seems like an isolated incident.

@rsc rsc closed this Dec 28, 2015

@mloskot

This comment has been minimized.

Copy link
Author

commented Dec 28, 2015

@rsc I understand your rationale and I don't aim to keep requesting any changes.

However, just to address some of your points

  • I found a public API which uses similar scheme (see my comment above about https://confluence.ucop.edu/display/Curation/ARK).
  • the current implementation of net/http does not completely conform to the RFC 3986 on normalization (again, see my previous comment).
@bradfitz

This comment has been minimized.

Copy link
Member

commented Jan 16, 2016

Amusingly, I just discovered that my Hunter Douglas Powerview blinds at home have this problem.

The blinds have a manual remote control that speaks some radio protocol, but there's also the "Powerview Hub" which puts the blinds on your wifi with an app. The Hub runs an HTTP server on port 80 speaking unauthenticated, unencrypted JSON.

But .... if you leave off the trailing question mark in certain queries, the hub crashes.

Another person who hit this: http://forum.universal-devices.com/topic/16538-hunter-douglas-powerview-control-with-isy/ who says:

3 - Whew, now time to get the SceneID's, open a browser, I used Chrome, enter http://(Your Hub IP)/api/scenes? (note: the ? is required or the hub will hang) If all goes well, you should get a JSON result like this: (Mine of course, yours will vary)

I'm reopening this bug, not because we'll necessarily change anything, but because it was closed for lack for external examples.

@bradfitz bradfitz reopened this Jan 16, 2016

@bradfitz bradfitz added this to the Unplanned milestone Jan 16, 2016

@rsc

This comment has been minimized.

Copy link
Contributor

commented Jan 17, 2016

I suppose we could, for Go 1.7, add a new field ForceQuery bool to url.URL, set it to true during url.Parse (only) when the URL ends in ? with no actual query string, and then change the printer to say if u.ForceQuery || u.RawQuery != "" instead of where it presumably today says if u.RawQuery != "".

@bradfitz bradfitz modified the milestones: Go1.7, Unplanned Jan 20, 2016

@gopherbot

This comment has been minimized.

Copy link

commented Feb 26, 2016

CL https://golang.org/cl/19931 mentions this issue.

@gopherbot gopherbot closed this in 9f26170 Feb 26, 2016

@bjkail

This comment has been minimized.

Copy link

commented Feb 29, 2016

It looks like this improperly parses URLs if the query string ends in ?, such as http://host/?q=?.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Feb 29, 2016

@bjkail, please file a new bug and reference this one.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.