Rewrite extra::url, based on the right spec

[SimonSapin]

The common fix for #8486, #10705, and #10706 is to rewrite the module based on the spec at http://url.spec.whatwg.org/ and the tests at https://github.com/w3c/web-platform-tests/tree/master/url

(Note: the tests assume the URLUtils API, designed for JavaScript in web browsers. We do not necessarily want the same API for Rust outside of testing code.)

[gsemet]

I'm interested in this

[SimonSapin]

@stibbons I have a work-in-progress implementation. I’ll publish it later today so you can pick it up if you want.

[SimonSapin]

Here it is: https://github.com/SimonSapin/rust-url
Note that I’m targeting Rust 67d7be0, the version used in Servo. (Though there is an upgrade being worked on.)

IDNA and the actual URL parser are not there yet, but everything else (host/domain/ipv6, application/x-www-form-urlencoded, percent encoding, test harness) is there. This should start being useful as soon as the URL parser is added.

The host parser returns an error on non-ASCII domains. This is good enough for now, IDNA support can be added later.

@jgraham also made an in-progress implementation: https://gist.github.com/SimonSapin/7794849
It has some known bugs (SeekableCharIterator is incorrect with non-ASCII input, at least) but the parser code can probably be reused.

[SimonSapin]

Also, be aware that the URL spec itself is still moving and has a number of open bugs:
https://www.w3.org/Bugs/Public/buglist.cgi?component=URL&list_id=30212&product=WHATWG&resolution=---

[SimonSapin]

I added Punycode encoding and decoding (with tests) to https://github.com/SimonSapin/rust-url but not Nameprep, which is the other big part of IDNA.

[SimonSapin]

rust-url is now (IMHO) ready to replace extra::url.

It is not "done" yet (some bugs remain, including in the spec!), but at this point it has more than feature-parity and is probably correct in more corner cases.

Given the current tendency to move things out of extra, rust-url probably shouldn’t move in. It could be distributed with rustc like the rest of the modules moving out of extra, but this has the downside of tying rust-url upgrades to compiler upgrades, and only weak advantages IMO.

So, whenever the Rust team decides it’s appropriate, I suggest removing extra::url and recommending rust-url as a replacement.

[alexcrichton]

I would generally be in favor of a liburl in the distribution for now. This would certainly not be a package in mozilla/rust long-term, but for now putting everything in this repo is the ad-hoc solution we have until we get a real package manager.

[SimonSapin]

In that case, I’d like to iterate a bit on rust-url (port rust-http and Servo to it, …) before it moves into the distribution.

Also, rust-url depends on rust-encoding, which I guess would have to move as well.

[SimonSapin]

I’d say rust-url is now in "beta" state. I have branches of rust-http and Servo using it:
https://github.com/SimonSapin/rust-http/compare/rust-url
https://github.com/SimonSapin/servo/compare/rust-url

It builds with Cargo. It depends on rust-encoding (which also builds with Cargo), but for something non-essential that could be deactivated if avoiding the additional dependency helps.

I’d like people to start using it and give feedback. I think the best way to do that is to get rid of the old liburl. I’ll go through the RFC process, but before that, @alexcrichton what would you prefer?

• Replace liburl with snapshots[*] of rust-url. You were in favor of this in February, but Cargo has reached alpha since then.
• Just remove liburl, and tell people to use rust-url through Cargo.
• Something else?

[*] In any case, I’d like to keep a separate rust-url repository that I can update in Servo independently of language upgrades.

[brson]

+1 on removing liburl. Thanks @SimonSapin

[brson]

Perhaps we can deprecate all of liburl for one release cycle.

[emberian]

This seems like a great place to let cargo shine.

[SimonSapin]

@brson Deprecating sounds reasonable. That means emitting warnings by default, right?

[alexcrichton]

@SimonSapin, awesome work! I agree with @cmr and @brson that I think this is where cargo shines, and we can probably just remove liburl entirely without re-adding rust-url. We can also try to grow more official support for it in terms of documentation and automation (not present currently).

We can mark liburl as #[deprecated] with a message to use rust-url instead which should ferry everyone along quickly.