net/url: URL.String URL encodes a valid IDN domain

[apriendeau]

What version of Go are you using (go version)?

1.9.3

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN="/Users/austin/Go/bin"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/austin/Go"
GORACE=""
GOROOT="/usr/local/opt/go/libexec"
GOTOOLDIR="/usr/local/opt/go/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/2m/fhmql6f10134630h6j10xgdh0000gn/T/go-build605073144=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you do?

url.String is URL escaping valid IDN domains and not rendering them correctly. I passed in https://✪.random.url/1234 to url.Parse then ran u.String() and it encodes the subdomain which is a Unicode character. This can be any Unicode character.

https://play.golang.com/p/Kwlrvoghy31

What did you expect to see?

https://✪.random.url/1234

What did you see instead?

https://%E2%9C%AA.random.url/1234

[agnivade]

According to the documentation -

String reassembles the URL into a valid URL string.

https://%E2%9C%AA.random.url/1234 is a perfectly valid string. When you put that in your browser, it will automatically convert to proper unicode and load the site. A better example is "http://ko.wikipedia.org/wiki/위키백과:대문". Run the program and enter the output string into your browser and see.

In your case, if you want to get the original string, you need to use u.Host and u.Path to get the unicode url.

[apriendeau]

Yes, it is valid but it not rendering it correctly as a string but I don't know if we should be escaping subdomains that are valid.

[rsc]

It seems to me that both https://✪.random.url/1234 and https://%E2%9C%AA.random.url/1234 are valid, and they are equivalent, so it is not a problem that u.String() returns the latter. And since it already does return the latter, which is again a correct answer, we should not gratuitously change to the former.

[bradfitz]

What's the motivation for this? If this is for display purposes to end users, maybe we could add a different stringification method?

[apriendeau]

In our use case, we want to display this value with the Unicode value to users who have domains in our system that are Unicode so we don't have to display https://xn--2ci.random.url/1234 which is meanless to them. I know this example is not the best, but this is legible to our foreign users who Unicode characters in their domains.

[agnivade]

If you want a fix right now, try this -

func main() {
	test := "https://✪.random.url/1234"
	u, err := url.Parse(test)
	if err != nil {
		panic(err)
	}
	p, _ := url.PathUnescape(u.String())
	fmt.Println(u.String(), p)
}

[apriendeau]

@agnivade thanks. That is a solution that works even if there is method or information exposed about this process in more detail in the documentation would be helpful so people know to have a workaround for this.

https://golang.org/pkg/net/url/#URL.String is thorough but it could be more clear.

[agnivade]

@apriendeau - So you are okay with a documentation update ?