net/http: update the IDNA implementation from x/text

[bradfitz]

Opening this bug for @mpvl to sanity check the new net/http implementation of IDNA.

It's all been committed. See func idnaASCII in net/http/request.go:

func idnaASCII(v string) (string, error) {
        if isASCII(v) {
                return v, nil
        }
        // The idna package doesn't do everything from                                                                                  
        // https://tools.ietf.org/html/rfc5895 so we do it here.                                                                        
        // TODO(bradfitz): should the idna package do this instead?                                                                     
        v = strings.ToLower(v)
        v = width.Fold.String(v)
        v = norm.NFC.String(v)
        return idna.ToASCII(v)
}

I'm not a domain expert.

In particular, do we do what Chrome and Firefox do? A user should be able to type in any case with any widths in their URL bar (or in an HTML file's <a href="....."> attribute) and they should all canonicalize the same way.

Thanks!

[mpvl]

First question: what standards to Chrome, Firefox and Safari implement?

According to local domain experts (one of the authors of UTS #46), most, if not all, browsers currently use UTS #46 which was introduced by The Unicode consortium as a transition standard between IDNA 2003 and IDNA 2008, minimizing the substantial incompatibility between them.

The UTS #46 standard provides (roughly) two versions: with and without compatibility processing. Browsers may be in different modes w.r.t. removing compatibility processing in the transition to IDNA 2008, but for the browsers I tested compatibly processing seems to be the norm.

Looking at the spec, it is not quite trivial to implement it, but also not too hard. I'll look into supporting UTS #46.

[gopherbot]

CL https://golang.org/cl/30392 mentions this issue.

[gopherbot]

CL https://golang.org/cl/30550 mentions this issue.

[mpvl]

The two CLs mentioned here are close to passing all tests defined in http://www.unicode.org/Public/idna/9.0.0/IdnaTest.txt. That's the easy part. Now comes the hard part: defining what to do with errors.

An error does not mean a domain name cannot be used. For example, a browser may decide to show a label as punycode, instead of Unicode, in case of some errors. An implementation may also add additional constraints based on confusables and cross-script spoofing, for example.

The policy for Chrome is described here: https://www.chromium.org/developers/design-documents/idn-in-google-chrome. This also gives a decent overview how other browsers behave.

We have to choose which policies we want to implement here. I would not pick something that is language-dependent. Other than that, something like Firefox (and what Chrome will be) seems the nicest to me. Safari's approach looks like its the easiest to implement.

[gopherbot]

CL https://golang.org/cl/30552 mentions this issue.

[gopherbot]

CL https://golang.org/cl/31273 mentions this issue.

[gopherbot]

CL https://golang.org/cl/31274 mentions this issue.