Hypothesis doesn't tell me when a rule never assumes something successfully

[radix]

I expect this program to raise an exception because the okay rule never successfully runs:

from hypothesis.stateful import RuleBasedStateMachine, Bundle, rule
from hypothesis import assume

class StatefulTests(RuleBasedStateMachine):
    state = Bundle('state')

    @rule(target=state)
    def whatever(self):
        return "Okay."

    @rule(initial=state)
    def okay(self, initial):
        assume(initial == "Not Okay.")

StatefulTests.TestCase().runTest()

[DRMacIver]

To summarize some discussion from IRC:

It's almost impossible to solve this perfectly. The problem is that for strategies with lots of rules, the chances are very high that at least one will not get properly exercised in a given run, which means that if you have a bunch of perfectly reasonable assumes some of them will probably fail to ever pass in a given run. This means that simply failing if any rule never passes its assumptions is error prone to say the least.

However not doing anything about this is a major usability issue, so I'm going to try to figure out how to make this better, but it's unlikely that it can be made perfect.

[Zac-HD]

Closed by the new @invariant decorator in 3.7 - use it to decorate checks that should be run after every step.

[DRMacIver]

I don't think so? The problem here is with rules that are not supposed to be run after every step, but should still sometimes run.

[Zac-HD]

I think emiting a warning for each precondition which was never satisfied would be more reasonable, since we notionally check each of them before each step.

master...Zac-HD:report-unsat-preconditions suggests to me that this should only run once, i.e. at the end of the generate phase, rather than at the end of each example. Otherwise it still tends to be flaky, as swarm testing makes it easy to never-satisfy a predicate that requires some other rule to have run.

[Zac-HD]

Closing this issue in favor of the list of observability ideas linked above 🙂