-
Notifications
You must be signed in to change notification settings - Fork 17.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
runtime: select is not fair #21806
Comments
This is not just using reflect. This similar program shows the same problem using an ordinary
|
It seems to be a problem with
with the line that was in 1.8:
then the program runs correctly. CC @josharian |
also not fair for select {
case v = <-in2:
case v = <-in1:
case v = <-in2:
case v = <-in1:
} the ratio is constantly larger than 1.08. The ratio for the following is even worse, about 2.0 select {
case v = <-in1:
case v = <-in1:
case v = <-in1:
case v = <-in1:
case v = <-in1:
case v = <-in1:
case v = <-in1:
case v = <-in1:
case v = <-in1:
case v = <-in1:
case v = <-in1:
case v = <-in1:
case v = <-in1:
case v = <-in1:
case v = <-in1:
case v = <-in1:
case v = <-in2:
case v = <-in2:
case v = <-in2:
case v = <-in2:
case v = <-in2:
case v = <-in2:
case v = <-in2:
case v = <-in2:
case v = <-in2:
case v = <-in2:
case v = <-in2:
case v = <-in2:
case v = <-in2:
case v = <-in2:
case v = <-in2:
case v = <-in2:
} |
The problem is one of correlation.
Outputs:
So if we get a Maybe if we did the feedback in the other direction (shift right, xor based on the low bit), it would help. Or maybe we have to go back to using modulo so we can depend on the low bits. |
@randall77 looks like it is flaw of LFSR. Old version were not better: var state uint32 = 1
func fastrand() uint32 {
fr := state
fr <<= 1
fr ^= uint32(int32(fr)>>31) & 0x88888eef
state = fr
return fr
}
func fastrandn(n uint32) uint32 {
return uint32(uint64(fastrand()) * uint64(n) >> 31)
}
I'm thinking what could be done with. |
What about using xorshift instead? var state uint32 = 2463534242
func fastrand() uint32 {
fr := state
fr ^= fr<<13
fr ^= fr>>17
fr ^= fr<<5
state = fr
return fr
} Outputs:
|
@rasky , xorshift is slower. Otherwise it is good because it gives almost same randomness for high and low bits. I have other variant for func fastrandn(n uint32) uint32 {
// some constants with random bit pattern
// (this are from hash32.go)
const m1 = 3168982561
const m2 = 3339683297
fr := fastrand()
fr ^= m1
fr *= m2
return uint32(uint64(fr) * uint64(n) >> 32)
} It also gives fair distribution
And it is faster than xorshift. |
Benchmark:
func fastrand() uint32 {
mp := getg().m
fr := mp.fastrand
mx := uint32(int32(fr)>>31) & 0xa8888eef
fr = fr<<1 ^ mx
mp.fastrand = fr
fr ^= 3168982561
fr *= 3339683297
return fr ^ (fr >> 16)
}
|
Is 1ns slower important for fastrand? Does it ever use a noticeable amount of cpu in a hot loop? Xorshift is among the fastest with good properties; you can come up with endless number of faster prngs with worse properties, but I fear we might end up in a similar bug some day. |
My variant is definitely good for all of |
I think, performance should be measured on all supported platforms. Something that is faster on intel, could be slower on Power or Arm. |
Yes, xorshift on ARM is basically 3 instructions thanks to barrel shift, hard to beat. I still think that 1ns in fastrand is not going to have any real world impact, though. |
By adapting xorshift+ to 32-bit integers, using a triplet picked up by the original xorshift paper, I tested this: func fastrand_xorshiftplus() uint32 {
// WARNING: untested for statistical properties
s0, s1 := state1, state0
s1 ^= s1 << 5
s1 = s1 ^ s0 ^ (s1 >> 17) ^ (s0 >> 13)
state0, state1 = s0, s1
return s0 + s1
} which is faster than the original fastrand on my i7 skylake:
Notice that the triplet choice can be totally wrong, I've just used it to test speed, but we should probably try BigCrush on it, and maybe select a different triplet. |
I supposed that s1 ^= s1 << a
s1 ^= s0 ^ (s1 >> b) ^ (s0 >> c) |
Looks like tripple [10, 13, 10] doesn't pass diehard. Probably, it was mistaken. Triplet [10, 10, 13] passes, but could not be trusted. |
I did take it from xorshift's pdf: [5,17,13] is one of them, valid for 32-bit numbers. The period of this xorshift+ 32-bit version (with two 32-bit words of state) should be 2^64-1. |
5,17,13 - is for 1*32 bit generator. |
shifts for |
Yes, I see, you're right. |
I had an experince with running xorshift framework to generate all correct triplets. |
I tried [23,3,24], it passes SmallCrush but not Crush:
(full log) |
I was mistaken for [10,13,10] - i got error in my test program. |
With xorshift64 (
xorshift
xorshift64+ is just a bit slower than xorshift64, but I don't think we need that "extra quality". I leave priority to open changset to @rasky. |
Change https://golang.org/cl/62530 mentions this issue: |
Reopening for backport to 1.9.1. |
@ianlancetaylor @randall77 should I backport the patch as-is or you prefer a simpler version (and if so, what would you simplify)? |
I like the CL as is. |
Change https://golang.org/cl/64193 mentions this issue: |
Fix for sema.go https://go-review.googlesource.com/c/go/+/68050 |
Change https://golang.org/cl/68050 mentions this issue: |
Before CL 62530 fastrand always returned non-zero value, and one condition in sema.go depends on this behavior. fastrand is used to generate random weight for treap of sudog, and it is checked against zero to verify sudog were inserted into treap or wait queue. Since its precision is not very important for correctness, lets just always set its lowest bit in this place. Updates #22047 Updates #21806 Change-Id: Iba0b56d81054e6ef9c49ffd293fc5d92a6a31e9b Reviewed-on: https://go-review.googlesource.com/68050 Reviewed-by: Austin Clements <austin@google.com> Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
If we cherry-pick CL 64193 then we must also cherry-pick CL 68050, because CL 64193 breaks a different property of fastrand. I'm pretty skeptical of wholesale-replacing the random number generator and then putting it into a point release when we just found a serious problem with it 9 days ago. Maybe for Go 1.9.2 we should just fix fastrandn to use a % again. The move away from % got this performance win:
I'm happy to give that back for Go 1.9.2. The uses of fastrandn elsewhere in the runtime also look like they would appreciate not having terrible correlations, so I don't think we should change only select's use. |
Closing this and remilestoning Go 1.10. The fix for Go 1.9.2 is #22253. |
What version of Go are you using (
go version
)?Does this issue reproduce with the latest release?
yes, on 1.9
What operating system and processor architecture are you using (
go env
)?GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/bchenebault/DEV/git-oab.si.fr.intraorange/BU900102/ceo-be"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build913166031=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
What did you do?
A Go select statement with reflect package function
reflect.Select(...)
should block until at least one of the cases can proceed and make a uniform pseudo-random choice. It appears that the behavior has changed since returned values are no longer pseudo-random in specific cases only.Our case is the following : 2 signaling channels preceding 2 "logic" channels
https://play.golang.org/p/0TiMtsRk03
But the same version with only one channel preceding the 2 others works
https://play.golang.org/p/sy4RjEEjdg
This case perfectly works on go1.8.3 and below.
What did you expect to see?
A pseudo random distribution
What did you see instead?
A strange distribution, surely not pseudo random
The text was updated successfully, but these errors were encountered: