crypto/sha512: on 32-bit platforms, panic trying to Sum after a large amount of data hashed

[jblebrun]

(Note: this is the same problem as in #29517, but in the crypto/sha512 package. Fix will be similar)

What version of Go are you using (go version)?

$ go version 1.11.1

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env

$ go env
GOARCH="amd64"

GOBIN=""

GOEXE=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="darwin"

GOOS="darwin"

GOPROXY=""

GORACE=""

GOROOT="/usr/local/Cellar/go/1.11.1/libexec"

GOTMPDIR=""

GOTOOLDIR="/usr/local/Cellar/go/1.11.1/libexec/pkg/tool/darwin_amd64"

GCCGO="gccgo"

CC="clang"

CXX="clang++"

CGO_ENABLED="1"

GOMOD=""

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/w6/9bjdntr539z1v138l_8mvmqshs5f5p/T/go-build219458045=/tmp/go-build -gno-record-gcc-switches -fno-common"

Note: the issue only occurs targeting architectures for which an int is 32 bits. I can easily trigger the problem by building with GOARCH=386.

What did you do?

Hash a large amount of data (length large than a platform int), save the hash using binary.Marshal, restore it using binary.Unmarshal, and then try to Sum it.

This play.golang.org link contains a simple example that triggers the issue:
https://play.golang.org/p/Xfs6W_SnKhb

When I run it there, the arch I see is amd64p32, so the issue occurs.

What did you expect to see?

I expect to be able to successfully get a checksum for the unmarshaled sha512 without panic.

What did you see instead?

Upon calling h.Sum(nil), I get a panic similar to:

panic: d.nx != 0

goroutine 1 [running]:
crypto/sha512.(*digest).checkSum(0x1147a410, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/usr/local/Cellar/go/1.11.1/libexec/src/crypto/sha512/sha512.go:324 +0x4a9
crypto/sha512.(*digest).Sum(0x1147a340, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/Cellar/go/1.11.1/libexec/src/crypto/sha512/sha512.go:293 +0x4e
main.main()
	gen.go:35 +0x408
exit status 2

Possible resolution

Fix cast in unmarshal code (as in #29517)

[gopherbot]

Change https://golang.org/cl/156277 mentions this issue: crypto/sha512: fix casting of d.nx in UnmarshalBinary