shuffle() modulo bias and extra iteration #57
TLDR; I think there are two bugs in the shuffling function. I suggest how to fix them. I also built a "sandbox" for testing shuffling functions.
I recently went on a deep-dive into shuffling functions to figure out why the beacon_chain one gets stuck in a loop. I have some findings which I will report here.
I believe there are two errors in the present implementation:
I'll give a quick intro to the the Fisher-Yates shuffle and modulo bias for some background. I found it interesting, so you might too.
This is a method of shuffling a list "introduced" by Richard Durstenfeld in 1964 and "popularized" by Donald E. Knuth in The Art of Computer Programming. Apparently it's similar to work done by Ronald Fisher and Frank Yates in 1938. More @ wikipedia.
History aside, it looks like this:
It seems that this is the primitive for the shuffling algorithm used in the spec, so lets assume this is what we're going to use. Seems optimal.
For background, modulo bias is introduced when using modulo to "fit" an random integer of a large range into a smaller range when the smaller range is not a clean multiple of the larger range. In other words,
As an example, consider a function
A method to fix this is to simply filter out all "biased" results, that is any results which are greater than
def filtered_rng(n): while True: x = rng() if x < rand_max - rand_max % n: break return x
The developer can now call
I propose the following code to fix these issues (the unused
def shuffle(lst, seed, config=DEFAULT_CONFIG): lst_count = len(lst) assert lst_count <= 16777216 o = [x for x in lst] source = seed i = 0 while i < (len(lst) - 1): source = blake(source) for pos in range(0, 30, 3): remaining = lst_count - i if remaining == 1: break m = int.from_bytes(source[pos:pos+3], 'big') rand_max = 16777216 - 16777216 % remaining if m < rand_max: replacement_pos = (m % remaining) + i o[i], o[replacement_pos] = o[replacement_pos], o[i] i += 1 return o
You can find this code here in a "shuffling sandbox" I made to test the conclusions I have come to.
First, to address error 1 (modulo-bias), the
Secondly, to address error 2 (unnecessary iteration), the
During the process of this, I made another shuffling implementation here. It is slower than the other implementations (due to abstraction overhead in Python), however it separates the logic into "shuffler" and "rng" components. This is useful in gaining an understanding of the fundamental structures involved.
There are a couple of potential speed-ups I thought of whilst doing this, will test them at some point.
Point (2) is not very important, it would not change the outcome of the shuffle. Worth noting for client devs.
As always, happy for feedback. Sorry about the essay.
The text was updated successfully, but these errors were encountered:
FYI, I played around with optimization 1 (only use the minimum amount of bits) and it doesn't seem to be feasible/sensible. The "optimization" ran orders of magnitude slower than the original version in Python.