You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A Go select statement with reflect package function reflect.Select(...) should block until at least one of the cases can proceed and make a uniform pseudo-random choice. It appears that the behavior has changed since returned values are no longer pseudo-random in specific cases only.
changed the title [-]reflect.Select(...) no longer pseudo-random selects[/-][+]runtime: reflect.Select(...) no longer pseudo-random selects[/+]on Sep 8, 2017
This is not just using reflect. This similar program shows the same problem using an ordinary select statement.
package main
import (
"fmt"
)
func main() {
in1 := make(chan int, 15000)
in2 := make(chan int, 15000)
for i := 0; i < 15000; i++ {
in1 <- 1
}
for i := 0; i < 15000; i++ {
in2 <- 2
}
out := make(chan int)
go func() {
c1 := make(chan struct{})
c2 := make(chan struct{})
for {
var v int
select {
case <-c1:
case <-c2:
case v = <-in2:
case v = <-in1:
}
out <- v
}
}()
var nb1, nb2 int = 0, 0
ConsumerLoop:
for {
select {
case d := <-out:
switch d {
case 1:
nb1++
if nb1 > 5000 { // stop consume. At this point, messages should be the equidistribution
break ConsumerLoop
}
case 2:
nb2++
default:
panic(d)
}
}
}
print(nb1, "\n")
print(nb2, "\n")
if !areSameOrder(nb1, nb2) {
fmt.Printf("Should have as many nb1 as nb2, got %d nb1 and %d nb2\n", nb1, nb2)
}
}
func areSameOrder(a, b int) bool {
ratio := float32(a) / float32(b)
return ratio < 1.3 && ratio > 0.7
}
So if we get a 0 from fastrandn(4), the next output is guaranteed to be a 0 or a 1.
I think this is fundamentally a problem with this RNG when N is a power of two, because the values we return are highly predictive of whether we end up xoring in the feedback or not.
Maybe if we did the feedback in the other direction (shift right, xor based on the low bit), it would help. Or maybe we have to go back to using modulo so we can depend on the low bits.
@rasky , xorshift is slower. Otherwise it is good because it gives almost same randomness for high and low bits.
I have other variant for fastrandn:
funcfastrandn(nuint32) uint32 {
// some constants with random bit pattern// (this are from hash32.go)constm1=3168982561constm2=3339683297fr:=fastrand()
fr^=m1fr*=m2returnuint32(uint64(fr) *uint64(n) >>32)
}
If we cherry-pick CL 64193 then we must also cherry-pick CL 68050, because CL 64193 breaks a different property of fastrand.
I'm pretty skeptical of wholesale-replacing the random number generator and then putting it into a point release when we just found a serious problem with it 9 days ago.
Maybe for Go 1.9.2 we should just fix fastrandn to use a % again. The move away from % got this performance win:
I'm happy to give that back for Go 1.9.2. The uses of fastrandn elsewhere in the runtime also look like they would appreciate not having terrible correlations, so I don't think we should change only select's use.
Activity
[-]reflect.select() no longer pseudo-random choices[/-][+]reflect.Select(...) no longer pseudo-random choices[/+][-]reflect.Select(...) no longer pseudo-random choices[/-][+]reflect.Select(...) no longer pseudo-random selects[/+][-]reflect.Select(...) no longer pseudo-random selects[/-][+]runtime: reflect.Select(...) no longer pseudo-random selects[/+][-]runtime: reflect.Select(...) no longer pseudo-random selects[/-][+]runtime: select is not fair[/+]ianlancetaylor commentedon Sep 8, 2017
This is not just using reflect. This similar program shows the same problem using an ordinary
selectstatement.ianlancetaylor commentedon Sep 8, 2017
It seems to be a problem with
fastrandn. If I replace this line in runtime/select.go:with the line that was in 1.8:
then the program runs correctly.
CC @josharian
zigo101 commentedon Sep 8, 2017
also not fair for
the ratio is constantly larger than 1.08.
The ratio for the following is even worse, about 2.0
randall77 commentedon Sep 8, 2017
The problem is one of correlation.
Outputs:
So if we get a
0fromfastrandn(4), the next output is guaranteed to be a0or a1.I think this is fundamentally a problem with this RNG when N is a power of two, because the values we return are highly predictive of whether we end up xoring in the feedback or not.
Maybe if we did the feedback in the other direction (shift right, xor based on the low bit), it would help. Or maybe we have to go back to using modulo so we can depend on the low bits.
randall77 commentedon Sep 8, 2017
CC @funny-falcon
funny-falcon commentedon Sep 8, 2017
@randall77 looks like it is flaw of LFSR. Old version were not better:
I'm thinking what could be done with.
rasky commentedon Sep 8, 2017
What about using xorshift instead?
Outputs:
(https://play.golang.org/p/sD9jmKGWJy)
funny-falcon commentedon Sep 8, 2017
@rasky , xorshift is slower. Otherwise it is good because it gives almost same randomness for high and low bits.
I have other variant for
fastrandn:It also gives fair distribution
And it is faster than xorshift.
But it uses multiplication, and doesn't fix cases when raw
fastrandused.https://play.golang.org/p/29n_4OfVQq
funny-falcon commentedon Sep 8, 2017
Benchmark:
fastrandin this way:20 remaining items
randall77 commentedon Sep 16, 2017
I like the CL as is.
gopherbot commentedon Sep 17, 2017
Change https://golang.org/cl/64193 mentions this issue:
[release-branch.go1.9] runtime: improve fastrand with a better generatorfunny-falcon commentedon Oct 4, 2017
Fix for sema.go https://go-review.googlesource.com/c/go/+/68050
gopherbot commentedon Oct 4, 2017
Change https://golang.org/cl/68050 mentions this issue:
runtime: fix using fastrand in sema.goruntime: fix using fastrand in sema.go
rsc commentedon Oct 13, 2017
If we cherry-pick CL 64193 then we must also cherry-pick CL 68050, because CL 64193 breaks a different property of fastrand.
I'm pretty skeptical of wholesale-replacing the random number generator and then putting it into a point release when we just found a serious problem with it 9 days ago.
Maybe for Go 1.9.2 we should just fix fastrandn to use a % again. The move away from % got this performance win:
I'm happy to give that back for Go 1.9.2. The uses of fastrandn elsewhere in the runtime also look like they would appreciate not having terrible correlations, so I don't think we should change only select's use.
rsc commentedon Oct 13, 2017
Closing this and remilestoning Go 1.10. The fix for Go 1.9.2 is #22253.