net/http: update the IDNA implementation from x/text

[bradfitz]

Opening this bug for @mpvl to sanity check the new net/http implementation of IDNA.

It's all been committed. See func idnaASCII in net/http/request.go:

func idnaASCII(v string) (string, error) {
        if isASCII(v) {
                return v, nil
        }
        // The idna package doesn't do everything from                                                                                  
        // https://tools.ietf.org/html/rfc5895 so we do it here.                                                                        
        // TODO(bradfitz): should the idna package do this instead?                                                                     
        v = strings.ToLower(v)
        v = width.Fold.String(v)
        v = norm.NFC.String(v)
        return idna.ToASCII(v)
}

I'm not a domain expert.

In particular, do we do what Chrome and Firefox do? A user should be able to type in any case with any widths in their URL bar (or in an HTML file's <a href="....."> attribute) and they should all canonicalize the same way.

Thanks!

[mpvl]

First question: what standards to Chrome, Firefox and Safari implement?

According to local domain experts (one of the authors of UTS #46), most, if not all, browsers currently use UTS #46 which was introduced by The Unicode consortium as a transition standard between IDNA 2003 and IDNA 2008, minimizing the substantial incompatibility between them.

The UTS #46 standard provides (roughly) two versions: with and without compatibility processing. Browsers may be in different modes w.r.t. removing compatibility processing in the transition to IDNA 2008, but for the browsers I tested compatibly processing seems to be the norm.

Looking at the spec, it is not quite trivial to implement it, but also not too hard. I'll look into supporting UTS #46.

[gopherbot]

CL https://golang.org/cl/30392 mentions this issue.

[gopherbot]

CL https://golang.org/cl/30550 mentions this issue.

[mpvl]

The two CLs mentioned here are close to passing all tests defined in http://www.unicode.org/Public/idna/9.0.0/IdnaTest.txt. That's the easy part. Now comes the hard part: defining what to do with errors.

An error does not mean a domain name cannot be used. For example, a browser may decide to show a label as punycode, instead of Unicode, in case of some errors. An implementation may also add additional constraints based on confusables and cross-script spoofing, for example.

The policy for Chrome is described here: https://www.chromium.org/developers/design-documents/idn-in-google-chrome. This also gives a decent overview how other browsers behave.

We have to choose which policies we want to implement here. I would not pick something that is language-dependent. Other than that, something like Firefox (and what Chrome will be) seems the nicest to me. Safari's approach looks like its the easiest to implement.

[gopherbot]

CL https://golang.org/cl/30552 mentions this issue.

[gopherbot]

CL https://golang.org/cl/31273 mentions this issue.

[gopherbot]

CL https://golang.org/cl/31274 mentions this issue.

[gopherbot]

CL https://golang.org/cl/32155 mentions this issue.

[mpvl]

Ideally the new package should live in net/idna, as managing Unicode dependencies for three repos seems more tedious and prone to security issues than doing so for two.
If not possible, it can replace the package in x/net/idna (for now).

[bradfitz]

@mpvl, so you want me to vendor x/text/internal/export/idna (and its dependencies, including bidi stuff?) into std now?

Or are you going to do it? Feel free to reassign if you want me to.

[bradfitz]

I'm going to punt to Go 1.9. I think the support we have now is sufficient for all but the corner cases.

[mpvl]

Yeah, I'll work on this. I'll do so by first updating the implementation in x/net. But let's wait till start of 1.9, as you said.

…

On Thu, Dec 1, 2016 at 7:42 PM Brad Fitzpatrick ***@***.***> wrote: I'm going to punt to Go 1.9. I think the support we have now is sufficient for all but the corner cases. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#17268 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGJZR3h9ujePgoTK9B-b-hdLZpt4rWKJks5rDxUhgaJpZM4KJSVp> .

[gopherbot]

CL https://golang.org/cl/37111 mentions this issue.