'+' in query part of urls

[ulf5]

Hello!

When parsing an url that contains + in the query part of the url, I would've expected it to be percent encoded. As it seems to be the behaviour of all the websites I have found, to decode plus to space that is. Why is this not the case? I did read the discussion of #416, but I'm afraid I didn't really understand the answer in it. Are the web services, i.e. google etc, not following the standard by decoding plus to space?

fn main() {
    use url::Url;
    println!("{}", Url::parse("http://google.com/?q=rust+url github").expect("").as_str());
}

Will print this url: http://google.com/?q=rust+url%20github and if you visit the link then "rust url github" will be the query entered into the google search box.
To clarify what I'm asking, why is the + in the above url not be encoded as %2B?

[SimonSapin]

As far as the URL parser and serialized are concerned, q=rust+url%20github as a whole is a single component of the URL called the query string. Like other in URL components, some characters are percent-encoded in query strings. It’s only when that query string interpreted in application/x-www-form-urlencoded format to obtain a sequence of key/value pairs that the + character is significant.

The relevant bit of spec is: https://url.spec.whatwg.org/#urlencoded-parsing

Example:

fn main() {
    let mut url = url::Url::parse("http://google.com/?q=rust+url github").expect("");
    println!("{:?}", url.query_pairs().collect::<Vec<_>>());
    url.query_pairs_mut().append_pair("foo", "bar baz");
    println!("{:?}", url.query());
    println!("{:?}", url.query_pairs().collect::<Vec<_>>());
}

(Playground)

Output:

[("q", "rust url github")]
Some("q=rust+url%20github&foo=bar+baz")
[("q", "rust url github"), ("foo", "bar baz")]

[ulf5]

So if I intend to have my query string interpreted by a webserver I need to add the query parameters as pairs? Is there a case where I would not want this?

[SimonSapin]

Sorry, I’m really not sure what you’re asking. What problem are you trying to solve? Are you interacting with one particular server, or building a general-purpose client? What kind of server?

In HTML terms, Url::parse is similar to <a href="…"> and query_pairs_mut is similar to <form method="get"><input name="…">.

[ulf5]

Yeah, I have a hard time putting it into words I guess. I am trying to make a command line tool that correctly encode/decodes urls.

If I'm getting this right, the web server is supposed to try to differentiate between requests that are get requests from a form and other get requests, and decode plus to spaces only in the case the request was from a form? And in the case of my earlier example, since google is expecting the requests to be generated by their search form they decode + to space.

[SimonSapin]

Yes, servers will usually replace + with in a query string, especially if interpreting it as list separated by & of name and value separated by =. But different servers might do some details differently.

I think another part the problem is that “encode” or “decode” are concepts that are really used for URLs. (And that’s even without going into what “correctly” means exactly…)

Back to my earlier question: what is the point of this command-line tool? What problem does it solve? In what situation would I want to use it?

[ulf5]

I'm not trying to say I'm making some extremely useful program or anything. But if you want a bit of background I have been in several situations when I've needed to loop through a bunch of values from e.g. a database, and either build urls by encoding the value and concatenating to some prefix to make urls. Or more recently I got a list of unencoded urls like "http://example.com/folder name/page?id=Rules for process" and wanted to encode do get requests to them. I have previously solved these situation by making specific python scripts for each situation. Now I'm trying to learn rust and I wanted to make something small that I might actually use, so that's why I wanted to make a tool I could pipe the urls to and get them encoded.

Yes, I'm realising now that this isn't as straightforward as I thought previously.

I might be missing other use cases for urls, but it seems to me that generally I'd want to encode + as %2B in the query string if I intend to use the url to do a http request, and I assume that the url in not encoded before.

[SimonSapin]

Thinking of URLs as “being encoded” v.s. not is probably not a useful concept, because other pieces of software typically do not make such a binary distinction (as far as I known). If you’re making HTTP requests, what matters is what bytes you send on the network and how the server at the other end interprets them.

[ulf5]

I think I'm getting it now, at least why the Url::parse-function can't encode it. It couldn't possibly know if the user wants the receiving web server to interpret the + as a space or a plus, it only turns the input string to a valid url.

Thank you for taking the time to answer.

[kanekv]

@SimonSapin where QUERY_ENCODE_SET (https://docs.rs/percent-encoding/1.0.1/percent_encoding/struct.QUERY_ENCODE_SET.html) is coming from? What is the purpose of that set? If it's supposed to encode URIs, then + is reserved character and should be encoded. Currently it is not in that set and I don't understand what was the logic for including characters in this set.

[ulf5]

@kanekv I think the purpose is to make the string into a valid query string, i.e. not encode it to preserve the actual characters when decoded.

[kanekv]

@ulf5 what is valid query string? What rfc does it follow? There is URI rfc https://tools.ietf.org/html/rfc3986 that lists reserved chars and it's not QUERY_ENCODE_SET. I wonder where set from QUERY_ENCODE_SET is defined (which rfc).

[ulf5]

@kanekv https://url.spec.whatwg.org/#url-parsing

If one of the following is true
byte is less than 0x21 (!)
byte is greater than 0x7E (~)
byte is 0x22 ("), 0x23 (#), 0x3C (<), or 0x3E (>)
byte is 0x27 (') and url is special

[kanekv]

@ulf5 cool, thanks

[nox]

Nothing to do here, closing.