x/net/idna: ToASCII() mangles long strings

[bnoordhuis]

What version of Go are you using (go version)?

go version go1.10.1 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/bnoordhuis/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/bnoordhuis/go"
GORACE=""
GOROOT="/usr/lib/go-1.10"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.10/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build029944234=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Contrived test case:

package main

import ("strings"; "golang.org/x/net/idna")

func main() {
	n := 65665
	s := strings.Repeat("x", n) + "\uFFFF"
	// python3 -c 'print(("x"*65665+"\uFFFF").encode("punycode")[65665:])'
	expected := "-1f303716a"
	actual, err := idna.ToASCII(s)
	if err != nil {
		panic(err)
	}
	actual = actual[len("xn--") + n:]
	if actual != expected {
		panic(actual) // prints "-qo7g", not "-1f303716a"
	}
}

What did you expect to see?

An error from idna.ToASCII() or the output that python3's punycode encoder produces.

What did you see instead?

$ go run t.go 
panic: -qo7g

goroutine 1 [running]:
main.main()
        /home/bnoordhuis/src/go/t.go:16 +0x165
exit status 2

The overflow check here does not seem to catch the case where the wraparound is big enough that delta + (m - n) * (h + 1) >= 0.

In the test case, it's 0 + (65535 - 128) * (65665 + 1) == 4295016062 % 2**32 == 48766.

Either the input should be rejected (reasonable, many other punycode encoders do) or it should use int64 arithmetic, like python3 does. Python may be an outlier; I'm not aware of other encoders that behave like that.