random_module() not working, or am I misunderstanding the documentation?

[HelloThisIsFlo]

Hi,

I would like to test a non-deterministic piece of code in a way that'd allow me to reproduce a failure. I thought the random_module() strategy would help me do that, but it doesn't seem to work as I expected.

---

Example

I would think that the following example would return different values of a and b for each iteration of the tests. From what I understood, the random_module() strategy is supposed to call random.seed(SEED) before the test (where SEED is actually rand_mod.seed in this example).

@given(random_module())
def test_random_module(rand_mod):
    a = random.randint(0, 100)
    b = random.randint(0, 100)
    print(f'a={a} b={b} | Expected seed: {rand_mod.seed} ')

But the result looks like this:

a=49 b=97 | Expected seed: 16859
a=49 b=97 | Expected seed: 26741
a=49 b=97 | Expected seed: 107
a=49 b=97 | Expected seed: 100
a=49 b=97 | Expected seed: 38947
a=49 b=97 | Expected seed: 14469
a=49 b=97 | Expected seed: 52059
a=49 b=97 | Expected seed: 8055
a=49 b=97 | Expected seed: 180
. . .

---

Quick Fix

A quick fix seems to be to generate integers() and then use that as a seed. I tested it seems it fits the reproducibility criteria. But obviously if I could use the random_module() strategy it'd be much more flexible and readable.

@given(integers())
def test_random_module_quick_fix(seed):
    deterministic_random = random.Random(seed)
    with patch('random.randint', deterministic_random.randint):
        a = random.randint(0, 100)
        b = random.randint(0, 100)

        print(f'QF: a={a} b={b} | Expected seed: {seed} ')

---

Questions

• Is this a bug? Or the intended behavior?
• If it is the intended behavior, what does the random_module() strategy actually do?
  I couldn't see any difference when using it vs not.

Thanks for the great work on the framework, besides the headache of this afternoon trying to figure out the random_module() thing, I found the rest of the framework incredibly simple to use. After a few minutes only I was able to significantly shrink down some particularly verbose tests. Great job, that's amazing 😍

[HelloThisIsFlo]

Quick Fix - UPDATE

I just found a cleaner version of the "quick fix":

@given(randoms())
def test_random_module_quick_fix(hypothesis_random):
    with patch('random.randint', hypothesis_random.randint):
        a = random.randint(0, 100)
        b = random.randint(0, 100)

        print(f'QF: a={a} b={b} | Expected seed: {hypothesis_random.seed} ')

But it still doesn't answer the question of whether or not it is normal for the random_module() strategy to behave the way it does :)

[DRMacIver]

I don't think you're missing anything! This looks like it's almost certainly a bug in Hypothesis. Just trying to figure out what's going on now. My guess is we broke this with some recent changes to how we handle random state and apparently don't have an adequate test to cover that we actually ever produce different random answers here. 😞

[DRMacIver]

Huh. Longer standing issue than I thought it was - this was actually introduced in May last year. Good spot!

[HelloThisIsFlo]

Oh wow! Now that was efficient!
I did try to revert to an earlier version, the one before the numpy random seeding was introduced, but had no luck.

Thanks for the fix @DRMacIver