IDNA support #119

valenting · 2015-07-29T02:37:17Z

I wrote a UTS 46 implementation based on http://intertwingly.net/projects/pegurl/idna.js
It needs a fast path for ascii codepoints, but I wanted to get some feedback on it first.

SimonSapin · 2015-07-29T02:59:21Z

make_idna_table.py

+# except according to those terms.
+
+
+# Run as: python make_idna_table.py idna_table.txt > src/idna_table.rs


Is idna_table.txt provided by Unicode? Where can it be found?

http://www.unicode.org/Public/idna/latest/IdnaMappingTable.txt

Please add this URL in a code comment.

SimonSapin · 2015-07-30T14:21:46Z

src/idna.rs

+use std::char;
+
+fn idna_mapped(mapping: &'static [u32]) -> Result<String, &'static str> {
+  let mut ret = "".to_string();


Please use four space indents, for consistency with other files.

SimonSapin · 2015-07-30T14:32:03Z

src/host.rs

            if !domain.is_ascii() {
-                Err(ParseError::NonAsciiDomainsNotSupportedYet)
-            } else if domain.find(&[
+                domain = match punycode::encode_str(&domain) {


Converting to punycode should be part of domain_to_ascii per http://www.unicode.org/reports/tr46/#ToASCII , and should happen on individual dot-separated labels, not the entire domain.

SimonSapin · 2015-07-30T19:04:58Z

src/host.rs

-                Err(ParseError::NonAsciiDomainsNotSupportedYet)
-            } else if domain.find(&[
+                domain = match punycode::encode_str(&domain) {
+                    Some(dom) => "xn--".to_string() + &dom,


I think that adding the xn-- prefix should also be part of domain_to_ascii, but UTS#46 has a bug: whatwg/url#53

SimonSapin · 2015-07-30T19:09:57Z

Could you also integrate the tests form http://www.unicode.org/Public/idna/latest/IdnaTest.txt ?

I’m considering rewriting the Python scripts to Rust and running them as Cargo build scripts to avoid having generated files in the repository. However in Gecko it’s not as easy if you’re not using Cargo. It would be a separate .rs file to be compiled for the host (does Gecko do cross-compilation?), run with an OUT_DIR environment variable, and have that run be a dependency of building the url crate, which should be done with the same environment variable.

How painful is it to do in Gecko? If it would make your like more difficult I can stick to Python and generated files in the repository.

valenting · 2015-07-30T19:32:03Z

It would be a bit harder to have the Gecko build system run the rust files at build time, but at the moment that's not a blocker. We can include the the generated rust-generated files just as we do with the python generated ones at the moment.

bors-servo · 2015-08-11T00:19:54Z

☔ The latest upstream changes (presumably #125) made this pull request unmergeable. Please resolve the merge conflicts.

bors-servo · 2015-09-03T12:41:58Z

☔ The latest upstream changes (presumably #129) made this pull request unmergeable. Please resolve the merge conflicts.


        IDNA support

Fix #112 Fix #119


        Add UTS46 tests


        IDNA: replace boolean parameters with a struct with named fields

… and fix values for URLs where true/false were swapped.


        Use a custom enum rather than strings for IDNA errors


        Add IDNA mapping table in source form to the repository.


        Use char literals in IDNA mapping.


        Refactor IDNA


        IDNA: Add DNS length check


        Implement most validitiy criteria

SimonSapin · 2015-12-05T23:37:28Z

See #152


        IDNA: Implement (most) bidi rules.

valenting · 2015-12-15T21:07:47Z

I've implemented the BIDI checks, and now all of the tests pass (except the contextJ tests, which client software isn't required to pass).
For combining_mark I call into unicode_normalization::char::canonical_combining_class - which I fear might not map correctly to General_Category=Mark. The unicode-normalization table generator has this data, but doesn't use it:
https://github.com/unicode-rs/unicode-normalization/blob/master/scripts/unicode.py#L92


        Change unicode-bidi dependency to version 0.2.3

SimonSapin · 2016-01-15T17:48:49Z

Other than a few nits I think this is ready to land. Thanks again!


        Use matches! macro and chars.skip(2). Remove unused error code.

SimonSapin · 2016-01-18T16:23:22Z

@bors-servo r+

SimonSapin · 2016-01-18T16:24:46Z

Not sure what’s happening with homu, but the Travis build for the PR push is green.


        Merge pull request #119 from valenting/idna_new

IDNA support

SimonSapin reviewed Jul 29, 2015
View changes

SimonSapin reviewed Jul 30, 2015
View changes

src/idna.rs Outdated

use std::char;

fn idna_mapped(mapping: &'static [u32]) -> Result<String, &'static str> {

let mut ret = "".to_string();

This comment has been minimized.

Sign in to view

SimonSapin Jul 30, 2015
Member

Please use four space indents, for consistency with other files.

SimonSapin reviewed Jul 30, 2015
View changes

Ms2ger assigned SimonSapin Sep 2, 2015

valenting and others added 9 commits Jul 29, 2015

IDNA support

fee35ca

Fix #112 Fix #119

Add UTS46 tests

ca0ff3c

IDNA: replace boolean parameters with a struct with named fields

727220a

… and fix values for URLs where true/false were swapped.

Use a custom enum rather than strings for IDNA errors

7bb230f

Add IDNA mapping table in source form to the repository.

cd24385

Use char literals in IDNA mapping.

ed30dca

Refactor IDNA

5a8d271

IDNA: Add DNS length check

a6599b9

Implement most validitiy criteria

Loading status checks…

c35d040

SimonSapin mentioned this pull request Dec 5, 2015

IDNA support #152

Merged

IDNA: Implement (most) bidi rules.

Loading status checks…

cb187a9

Change unicode-bidi dependency to version 0.2.3

Loading status checks…

9801e28

SimonSapin mentioned this pull request Jan 15, 2016

Combining mark check in IDNA #158

Closed

Use matches! macro and chars.skip(2). Remove unused error code.

Loading status checks…

eb3846d

SimonSapin added a commit that referenced this pull request Jan 18, 2016

Merge pull request #119 from valenting/idna_new

Loading status checks…

d6e1ac5

IDNA support

servo / rust-url

IDNA support #119

IDNA support #119

valenting commented Jul 29, 2015

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

SimonSapin commented Jul 30, 2015

valenting commented Jul 30, 2015

bors-servo commented Aug 11, 2015

bors-servo commented Sep 3, 2015

SimonSapin commented Dec 5, 2015

valenting commented Dec 15, 2015

SimonSapin commented Jan 15, 2016

SimonSapin commented Jan 18, 2016

SimonSapin commented Jan 18, 2016

		# except according to those terms.


		# Run as: python make_idna_table.py idna_table.txt > src/idna_table.rs

servo / rust-url

Join GitHub today

IDNA support #119

IDNA support #119

Conversation

valenting commented Jul 29, 2015

This comment has been minimized.

SimonSapin Jul 29, 2015

This comment has been minimized.

valenting Jul 29, 2015

This comment has been minimized.

SimonSapin Jul 30, 2015

This comment has been minimized.

SimonSapin Jul 30, 2015

This comment has been minimized.

SimonSapin Jul 30, 2015

This comment has been minimized.

SimonSapin Jul 30, 2015

SimonSapin commented Jul 30, 2015

valenting commented Jul 30, 2015

bors-servo commented Aug 11, 2015

bors-servo commented Sep 3, 2015

SimonSapin commented Dec 5, 2015

valenting commented Dec 15, 2015

SimonSapin commented Jan 15, 2016

SimonSapin commented Jan 18, 2016

SimonSapin commented Jan 18, 2016