Allow non-UTF8 keys and value params in the query string

[95th]

Currently the API only allow writing or parsing rust strings which means enforced UTF-8. However the standard does not enforce this.

Some protocols (For e.g. Bittorrent) rely of sending bytes in a GET request. These bytes may or may not make a valid UTF-8 string.

Is it possible to provide methods to URL encode/decode name/value bytes pair?

[SimonSapin]

The Rust String and &str types happen to be represented in memory as UTF-8, but that’s really an implementation detail. You should think of them as Unicode strings, a separate type of data from byte strings. Converting a value between those types requires choosing an encoding.

Since you mention name/value pairs, I assume you mean not just any query string but one with application/x-www-form-urlencoded. In that format as specified in the WHATWG URL Standard, names and values are Unicode. That part is not gonna change. However, that algorithm also takes an encoding as a parameter.

Now if you pick ISO-8859-1 as the encoding (the variant that maps all bytes to Unicode code points of the same value, not Windows-1252), then you can smuggle arbitrary bytes to be percent-escaped by that algorithm.

In version 2.x, rust-url doesn’t depend on any character encoding library so you have to provide the encoding implementation to use something other than UTF-8:

use std::borrow::Cow;
use std::convert::TryFrom;

fn main() {
    println!(
        "{}",
        url::Url::parse("https://example.com")
            .unwrap()
            .query_pairs_mut()
            .encoding_override(Some(&iso_8859_1_encode))
            .append_pair("foo", &iso_8859_1_decode(b"\x00\xFF"))
            .finish()
    );
}

fn iso_8859_1_decode(bytes: &[u8]) -> String {
    bytes.iter().map(|&byte| char::from(byte)).collect()
}

fn iso_8859_1_encode(string: &str) -> Cow<[u8]> {
    string.chars()
        .map(|c| u8::try_from(u32::from(c)).unwrap())
        .collect()
}

(Playground)

Output:

https://example.com/?foo=%00%FF

[djc]

I think @SimonSapin's answer here is pretty definitive. Let me know if you have further questions/feedback.