math/rand: index out of range in (*rngSource).Int63

[erikdubbelboer]

I'm not sure if reporting this is useful as I can't replicate this at all. Maybe it was just a cosmic ray or maybe someone can use the stack trace to find some obscure bug.

What version of Go are you using (go version)?

go version go1.8.3 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/erik"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build313906869=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

Run a program that receives around 5000 requests per second using https://github.com/valyala/fasthttp

What did you expect to see?

No crashes

What did you see instead?

A very unusual panic in the Go source that I haven't seen before:

panic: runtime error: index out of range

goroutine 471249678 [running]:
math/rand.(*rngSource).Int63(0xc427177500, 0x0)
        /usr/local/go/src/math/rand/rng.go:231 +0x8c
math/rand.(*Rand).Int63(0xc4260b7fb0, 0x0)
        /usr/local/go/src/math/rand/rand.go:81 +0x33
math/rand.(*Rand).Int63-fm(0x0)
        /usr/local/go/src/math/rand/rand.go:211 +0x2a
math/rand.read(0xc46fdd8500, 0x100, 0x100, 0xc5a8135088, 0xc4260b7fd0, 0xc4260b7fd8, 0xc46fdd8500, 0x100, 0xc5a8135120)
        /usr/local/go/src/math/rand/rand.go:219 +0x9a
math/rand.(*Rand).Read(0xc4260b7fb0, 0xc46fdd8500, 0x100, 0x100, 0x20, 0xc46c04bde0, 0x0)
        /usr/local/go/src/math/rand/rand.go:211 +0x110
io.ReadAtLeast(0xf83d40, 0xc4260b7fb0, 0xc46fdd8500, 0x100, 0x100, 0x100, 0xb76620, 0x1, 0xc46c04bde0)
        /usr/local/go/src/io/io.go:307 +0xa9
io.ReadFull(0xf83d40, 0xc4260b7fb0, 0xc46fdd8500, 0x100, 0x100, 0x100, 0xc8b1fd730f65f11f, 0xd6990624d192e819)
        /usr/local/go/src/io/io.go:325 +0x58
crypto/rand.Int(0xf83d40, 0xc4260b7fb0, 0xc42773fb60, 0x410f02, 0xc468f70ea0, 0x120)
        /usr/local/go/src/crypto/rand/util.go:122 +0x14a
crypto/rsa.decrypt(0xf83d40, 0xc4260b7fb0, 0xc4276bf320, 0xc5a8135430, 0x0, 0x0, 0x0)
        /usr/local/go/src/crypto/rsa/rsa.go:499 +0x5f7
crypto/rsa.decryptAndCheck(0xf83d40, 0xc4260b7fb0, 0xc4276bf320, 0xc5a8135430, 0xc5a8135430, 0x100, 0x0)
        /usr/local/go/src/crypto/rsa/rsa.go:559 +0x56
crypto/rsa.SignPKCS1v15(0xf83d40, 0xc4260b7fb0, 0xc4276bf320, 0x5, 0xc46c04bdc0, 0x20, 0x20, 0x40ef28, 0xb62060, 0xc4852c0c58, ...)
        /usr/local/go/src/crypto/rsa/pkcs1v15.go:250 +0x2bb
crypto/rsa.(*PrivateKey).Sign(0xc4276bf320, 0xf83d40, 0xc4260b7fb0, 0xc46c04bdc0, 0x20, 0x20, 0xf84bc0, 0xc4852c0c58, 0x0, 0x0, ...)
        /usr/local/go/src/crypto/rsa/rsa.go:105 +0x167
crypto/tls.(*ecdheKeyAgreement).generateServerKeyExchange(0xc42cd72e10, 0xc425fa4300, 0xc4276bf440, 0xc4302eb040, 0xc47be936c0, 0x10d8, 0x0, 0x0)
        /usr/local/go/src/crypto/tls/key_agreement.go:276 +0x565
crypto/tls.(*serverHandshakeState).doFullHandshake(0xc5a8135a58, 0xc5a8135a00, 0x0)
        /usr/local/go/src/crypto/tls/handshake_server.go:404 +0x3f3
crypto/tls.(*Conn).serverHandshake(0xc4f9914e00, 0xc4aa08, 0xc4f9914f20)
        /usr/local/go/src/crypto/tls/handshake_server.go:85 +0x325
crypto/tls.(*Conn).Handshake(0xc4f9914e00, 0x0, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:1309 +0x2db
crypto/tls.(*Conn).Read(0xc4f9914e00, 0xc55b8d22c8, 0x1, 0x1, 0x0, 0x0, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:1117 +0x59
github.com/erikdubbelboer/fasthttp.acquireByteReader(0xc5a8135e48, 0xf908c0, 0xc4f9914e00, 0xc438725b00)
        /home/erik/src/github.com/erikdubbelboer/fasthttp/server.go:1807 +0x146
github.com/erikdubbelboer/fasthttp.(*Server).serveConn(0xc4277c9cc0, 0xf908c0, 0xc4f9914e00, 0xc42739c901, 0xc427390101)
        /home/erik/src/github.com/erikdubbelboer/fasthttp/server.go:1497 +0x156d
github.com/erikdubbelboer/fasthttp.(*Server).(github.com/erikdubbelboer/fasthttp.serveConn)-fm(0xf908c0, 0xc4f9914e00, 0xc5a8135f78, 0x1)
        /home/erik/src/github.com/erikdubbelboer/fasthttp/server.go:1273 +0x3e
github.com/erikdubbelboer/fasthttp.(*workerPool).workerFunc(0xc42739c980, 0xc437cb3160)
        /home/erik/src/github.com/erikdubbelboer/fasthttp/workerpool.go:210 +0xde
github.com/erikdubbelboer/fasthttp.(*workerPool).getCh.func1(0xc42739c980, 0xc437cb3160, 0xaf1f40, 0xc437cb3160)
        /home/erik/src/github.com/erikdubbelboer/fasthttp/workerpool.go:182 +0x35
created by github.com/erikdubbelboer/fasthttp.(*workerPool).getCh
        /home/erik/src/github.com/erikdubbelboer/fasthttp/workerpool.go:184 +0x135

[ALTree]

The out-of-bounds must be on rng.vec in rng.go (the stack trace says line 231 because Uint64() is inlined there).

rng.vec is only accessed using rng.tap and rng.feed, which are only changed by decrementing them, and when they become negative we add _LEN (which is the size of rng.vec).

rng.tap--
if rng.tap < 0 {
    rng.tap += _LEN
}

So when we use them to access the vector, both rng.tap and rng.feed can't be negative and can't be bigger than _LEN - 1. I don't see where the out-of-bounds panic could come from.

Considering you can't reproduce this, it smells like memory corruption (possibly caused by a data race).

[erikdubbelboer]

I didn't look at the code yet but I agree that it shouldn't be possible to have an out-of-bounds panic here.

I don't think it's in a data race in our process as that would normally make it happen much more of then than this once in a year.

This was on a Google Compute instance which wasn't doing anything else. Not sure how susceptible these are to memory corruptions from something else but I guess it can happen.

[ALTree]

It's also possible that the memory corruption was caused by a bug in the Go runtime.

This issue, thought, as it is, it's not actionable: the stack trace tells us nothing, and you can't reproduce the crash, so we cannot use your report to extract a reproducer for a (possible) runtime bug.

Leaving this open in case someone else want to say something about this.

[bradfitz]

fasthttp plays games with unsafe and sharing memory between []byte and string so my appetite for debugging this is minimal.

[erikdubbelboer]

I'm not sure the unsafe behavior of fasthttp is to blame, I would expect it to happen much more often in this case. I think it was just some random memory corruption so it will be impossible to debug.

[erikdubbelboer]

I found my mistake. As you can see from the stack trace I am using tls but I supplied it with math/rand.Source which is not thread safe. tls requires a safe random source.

[ALTree]

@erikdubbelboer thanks for the update!