net/url: encoding inconsistency between 1.4 and 1.5 for unicode domain names

[VojtechVitek]

Given this code (Go Playground):

package main

import (
    "fmt"
    "net/url"
)

func main() {
    u, _ := url.Parse("http://www.žluťoučký-kůň.cz")
    fmt.Println(u)
}

I get different results in go 1.4 and go 1.5:

$ go version
go version go1.4.2 darwin/amd64

$ go run main.go 
http://www.žluťoučký-kůň.cz

$ go version
go version go1.5.1 darwin/amd64

$ go run main.go 
http://www.%C5%BElu%C5%A5ou%C4%8Dk%C3%BD-k%C5%AF%C5%88.cz

Is this intended (but undocumented) behavior - or is it a bug?

[rakyll]

I assume the conversion done to maximize the interoperability with the legacy URI resolvers, but the RFC 3987 particularly recommends replacing non-ascii with dashes primarily.

From http://www.ietf.org/rfc/rfc3987.txt,

   The ToASCII operation may fail, but this would mean that the IRI
   cannot be resolved.  This conversion SHOULD be used when the goal is
   to maximize interoperability with legacy URI resolvers.  For example,
   the IRI

   "http://résumé.example.org"

   may be converted to

   "http://xn--rsum-bpad.example.org"

   instead of

   "http://r%C3%A9sum%C3%A9.example.org".

/cc @rsc @bradfitz

[VojtechVitek]

Well, there's already http://godoc.org/golang.org/x/net/idna#ToASCII for Punycode conversions.

[rakyll]

In any case, this is a bug. We should never over escape the host name because url.Parse doesn't allow parsing percent-encoded host names now.

u, _ := url.Parse("http://www.žluťoučký-kůň.cz")
if _, err := url.Parse(u.String()); err != nil {
  panic(err)
}

will panic with "panic: parse http://www.%C5%BElu%C5%A5ou%C4%8Dk%C3%BD-k%C5%AF%C5%88.cz: percent-encoded characters in host".

[rsc]

RFC 3986 is clear about the need for percent-encoding the host when creating the URL. The parser is a bit lax in accepting the UTF-8 to begin with, but it's probably a mistake to reject it at this point.

I sent a CL to accept the %-encoded form so that we can round-trip the URL. I'm not sure this is a great idea, but we'll see I guess. The main argument is for non-HTTP uses of URLs.

[gopherbot]

CL https://golang.org/cl/17385 mentions this issue.