math/rand: After ~1.5 month of valid usage, rand started consistently returning 0

[bjg2]

What version of Go are you using (go version)?

1.17.1

Does this issue reproduce with the latest release?

Can't say, don't know how to reproduce.

What operating system and processor architecture are you using (go env)?

It was built for amd64 and linux, and running in alpine (docker in k8s).

What did you do?

I was pretty usually using math.rand, with the following code:

var seededRand *rand.Rand = rand.New(rand.NewSource(time.Now().UnixNano()))

const charset = "abcdefghijklmnopqrstuvwxyz"

func RandomString(length int) string {
	b := make([]byte, length)
	charsetLen := len(charset)
	for i := range b {
		b[i] = charset[seededRand.Intn(charsetLen)]
	}
	return string(b)
}

What did you expect to see?

Random sting, as I usually did.

What did you see instead?

I had 3 instances of this service running. After the long time of running as expected (~1.5 months) one service started consistently returning "aaaaaaaaaa" for RandomString(10) (other 2 continued working as expected).

This was a very weird situation, would someone know what might have happened, is it fixed in some new go version, or how to mitigate? I'm thinking of mitigating by changing to a new random seed if I recognize the random returning always 0....

[randall77]

This is strange. I don't see how the math/rand code could cause this.

Possibly there is a memory management bug that causes seededRand.src.(*rngSource).vec to be zeroed. Then it would return all zeros from then on.

Without a way to reproduce this is going to be hard to track down. If my suspicion is right, either it is a bug in the runtime, or the kernel, or some bad use of unsafe. It isn't in the rand code at all.

[zephyrtronium]

Of note, the only way for the algorithm math/rand uses to return more than 606 consecutive zeroes is for the entire state to be zero. No state with any nonzero element will ever become all zero on its own. So, I agree that the bug is elsewhere.

[bjg2]

Not sure if important for investigation, but rand does not return 0 but x where x %26 == 0 (using intn). Also, function returned "aaaaaaaaaa" at least 360 times in a row, which means it generated x where x % 26 == 0 at least 3600 times in a row.

[randall77]

Not sure if important for investigation, but rand does not return 0 but x where x %26 == 0 (using intn).

I don't understand your point here. Could you elaborate?

[bjg2]

Code that caused issues uses Intn(26), and was always returning 0 after some point in time. I guess that internally Intn(26) is something like randInt() % 26. I guess that correct claim would be that randInt() does not consistently returns 0 but consistently returns numbers divisible by 26.

@zephyrtronium mentined that rand algorithm cannot return more than 606 consecutive zeros, but it does not have to do that to cause the reported issue. It has to enter into some weird state in which it always returns numbers divisible by 26.

[randall77]

I see. You're just saying you merely observed Intn(26) returning 0s, but you don't know whether the internal (entire 32 or 64 bit) RNG was returning 0s.
I think it is very likely that if we're returning lots of numbers y=x%26 such that y==0, then internally we're generating lots of numbers such that x==0. There's nothing magical inside the RNG related to the number 26 that would cause lots of xs to be 0 mod 26 but not be all 0.

[slrz]

Note that above RandomString function is not safe to invoke concurrently from multiple goroutines. Did your service account for that and serialized the calls to it?

[bjg2]

What's not safe about it, rand.Rand's Intn() is not routine-safe?

I do call it from multiple routines concurrently without serialization. Could that have caused the issue?

[zephyrtronium]

Per the package documentation on math/rand: "The default Source is safe for concurrent use by multiple goroutines, but Sources created by NewSource are not." Based on how the generator updates the feed and tap, it seems likely that this is a sufficient explanation.

[bjg2]

Well thank you very much then! I will sync it and it will hopefully be good forever. :) Thank you very much again, good catch!

General language feedback: though documentation is important to read, devs often will not do that, especially for simple, 'everyday' stuff. It would be interesting to think whether some tooling would be able to help dev here, func annotation, warning, etc.

[randall77]

Running under the race detector should catch the issue described.