net/http: use nontransitional IDNA processing

[TimothyGu]

What version of Go are you using (go version)?

$ go version
go version go1.16.3 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/timothy-gu/.cache/go-build"
GOENV="/home/timothy-gu/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/timothy-gu/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/timothy-gu/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.3"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/tmp/gourl/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build343371698=/tmp/go-build -gno-record-gcc-switches"

What did you do?

package main

import (
	"fmt"
	"net/http"
)

func main() {
	resp, _ := http.Get("http://straße.de/")
	fmt.Println("straße", resp.Header.Get("Content-Length"))

	resp, _ = http.Get("http://strasse.de/")
	fmt.Println("strasse", resp.Header.Get("Content-Length"))

	resp, _ = http.Get("http://xn--strae-oqa.de/")
	fmt.Println("xn--strae-oqa", resp.Header.Get("Content-Length"))
}

What did you expect to see?

straße.de should get resolved to xn--strae-oqa per IDNA Nontransitional processing and DENIC announcement. This is the case on Firefox and Safari when you type the URL into the address bar. It is also the behavior of curl, wget, Node.js, and the browser WHATWG URL Standard.

straße 33560
strasse 6637
xn--strae-oqa 33560

What did you see instead?

straße.de got resolved instead to strasse.de. This is the current behavior in Chrome, and from what Ryan Sleevi implied in https://crbug.com/694157, other pieces of Google software as well. (The same behavior is in Python as well, but that's a bit unfair, since Python's IDNA encoding is effectively frozen to IDNA 2003.)

straße 6637
strasse 6637
xn--strae-oqa 33560

Discussion

Which way to go here is ultimately a policy change. The arguments for keeping transitional processing are:

• Maintains strict compatibility with previous versions of Go
• Maintains compatibility with Chrome, and Google products in general

The arguments for moving to nontransitional processing are:

• Aligns with the rest of the world
• Aligns with web standards

It appears to me that implementing things from web standards and first principles is part of the Go ethos, which would lean in favor of using nontransitional processing. However, maintaining compatibility is also a powerful motivation.

[seankhliao]

cc @bradfitz @rsc @empijei

[networkimprov]

@ianlancetaylor possible proposal?

[ianlancetaylor]

I'll let net/http maintainers comment.

[networkimprov]

cc @fraenkel @neild

[neild]

Empirical testing of navigating to http://straße.de/ in various browsers (version whatever was installed on the machine closest to me at the time):

• Chrome: strasse.de
• Safari: xn--strae-oqa.de
• Firefox: xn--strae-oqa.de
• Edge: strasse.de

My most-convenient Linux box has curl 7.74.0 with IDNA support and resolves http://straße.de/ to xn--strae-oqa.de. I also found CVE-2016-8625 for libcurl, describing the use of transitional processing in libcurl before version 7.51.0 as a security vulnerability.

There appears to be no consistency in browser behavior.

However, given that:

• the .de registry has been issuing domains containing ß for about a decade
• many browsers (and client libraries like libcurl) do use nontransitional processing
• nontransitional processing is the desired end state

it seems reasonable to me to change the "golang.org/x/net/idna".Lookup profile default to use nontransitional processing.

Marking this as a proposal.

cc @FiloSottile for possible security considerations.

[FiloSottile]

I wouldn’t go as far as calling the current behavior a security vulnerability, but unexpected and spec-divergent behavior is a security liability, so I generally support the change.

This fits in a pattern of implementation choices that haven’t kept up with the changing ecosystem, to the point of becoming dangerously idiosyncratic (like most recently semicolons in URL queries). We don’t have an exception for them in the Compatibility Promise, but maybe we should discuss adding one to ensure the Go standard library doesn’t become riddled with legacy surprising behaviors.

By the way, if we make this change, we should document not only the new behavior but also how it will change and will be kept up to date.

[rsc]

What is Chrome's rationale for not making this change?

[favonia]

What is Chrome's rationale for not making this change?

FYI: There have been multiple issues on Chromium over the years about transitional v.s. non-transitional processings. The latest issue (that was not merged, blocked, or closed) is https://bugs.chromium.org/p/chromium/issues/detail?id=694157 There was an older one closed as WONTFIX: https://bugs.chromium.org/p/chromium/issues/detail?id=505262

[neild]

I have not been able to find a conclusive rationale for Chrome not making this change. That doesn't mean there isn't one, of course, just that I can't find it.

So far as I can tell, the primary concern that led to the introduction of transitional processing is that the case-insensitivity of domain names introduces an ambiguity: Uppercase ß is SS, so uppercasing straße.de and strasse.de ambiguously produce STRASSE.DE.

Of course, the current situation is that lowercase straße.de produces ambiguous results depending on which HTTP client you use, so avoiding ambiguity entirely does not presently appear to be an option.

By the way, if we make this change, we should document not only the new behavior but also how it will change and will be kept up to date.

We currently document that the idna.Lookup and idna.Display profiles may change over time, so I believe that a change to nontransitional processing would be within the scope of our documented compatibility promises.

I don't know what a good policy for future changes would be. It seems to me that the lack of a clearly correct policy has led to the current fragmentation between browsers. A simple and clear one for us would be to defer the decision to some other party--e.g., always do what Chrome does by default.