Way to start a recheck from a fully shrunk case?

[altxt]

In some cases, it can be beneficial to inspect in a debugger the behavior of the function under test. If it is sufficiently complex, this is better done using the input data that is as simple as possible - i.e. shrunken. Debugging is often done by seting breakpoints, so it is reasonable to expect the breakpoints to be hit starting directly with the shrunk unput. But I was unable to find a way of making it so. The reproduction info that Hedgehog prints seems to lead to the failing case that was found first, not to the result of shrinking. Am I missing something?

Here is the repro in code. We check a function with clear failure conditions and we log both the input and the number of calls:

[Test]
public void Hedgehog_Recheck_StartsDirectlyFromShrunkenCase()
{
    int count = 0;
    // any input in [13 .. 20] range fails the test, with 13 being the simplest failing case
    Func<int, bool> testFunc = x =>
    {
        count++;
        Console.WriteLine($"x = {x}, count = {count}");
        return x < 13 || x > 20;
    };
    var property =
        from x in Property.ForAll(Gen.Int32(Range.Constant(1, 1000)))
        select testFunc(x);
    
    // for getting the seed
    //property.Check(PropertyConfig.Default.WithShrinks(100));

    // for the actual test
    property.ReportRecheck(41, new Seed(15041786197103353380UL, 17636625472901432099UL), PropertyConfig.Default.WithTests(1));
    Assert.AreEqual(1, count, "Function under test called more than once");
}

Initial run, getting the seed for recheck:

System.Exception : *** Failed! Falsifiable (after 41 tests and 1 shrink):
13
This failure can be reproduced by running:
> Property.recheck (41 : Size) ({ Value = 15041786197103353380UL; Gamma = 17636625472901432099UL }) <property>

   at Hedgehog.ReportModule.tryRaise(Report report)
   at HedgehogSimpleTests.Hedgehog_Recheck_StartsDirectlyFromShrunkenCase()

x = 900, count = 1
x = 897, count = 2
x = 503, count = 3
x = 513, count = 4
x = 557, count = 5
x = 440, count = 6
x = 197, count = 7
x = 264, count = 8
x = 308, count = 9
x = 121, count = 10
x = 665, count = 11
x = 453, count = 12
x = 225, count = 13
x = 610, count = 14
x = 116, count = 15
x = 983, count = 16
x = 739, count = 17
x = 388, count = 18
x = 989, count = 19
x = 157, count = 20
x = 257, count = 21
x = 600, count = 22
x = 950, count = 23
x = 359, count = 24
x = 22, count = 25
x = 419, count = 26
x = 831, count = 27
x = 982, count = 28
x = 187, count = 29
x = 862, count = 30
x = 271, count = 31
x = 113, count = 32
x = 442, count = 33
x = 420, count = 34
x = 35, count = 35
x = 96, count = 36
x = 387, count = 37
x = 592, count = 38
x = 986, count = 39
x = 253, count = 40
x = 14, count = 41
x = 1, count = 42
x = 7, count = 43
x = 11, count = 44
x = 13, count = 45
x = 1, count = 46
x = 7, count = 47
x = 10, count = 48
x = 12, count = 49

Test run, see if we hit a shrunk case or just any failing case on the first invocation:

 Function under test called more than once
  Expected: 1
  But was:  9

   at HedgehogSimpleTests.Hedgehog_Recheck_StartsDirectlyFromShrunkenCase()

x = 14, count = 1
x = 1, count = 2
x = 7, count = 3
x = 11, count = 4
x = 13, count = 5
x = 1, count = 6
x = 7, count = 7
x = 10, count = 8
x = 12, count = 9

[TysonMN]

The reproduction info that Hedgehog prints seems to lead to the failing case that was found first, not to the result of shrinking. Am I missing something?

You are not missing anything. That is the intended behavior.

The idea of directly rechecking the shrunken case is interesting though. I will think about that and also test your provided code later.

[TysonMN]

We could create a bit vector that says if each input in the shrinking process passes the test (1) or fails the test (0). Then we could use this bit vector to skip running each test and find the shrinken input. With the shrinken input in hand, we would run the test once more for the user to debug it.

Now that I have this idea, I am excited to implement it. Not sure at the moment when I will have time though.

[TysonMN]

I wrote some code in an attempt to solve this problem, but I can't figure out how to avoid executing the test while traversing the shrink tree.

Can anyone explain how this might be done?

[TysonMN]

jetCheck, a PBT library for Java, has this feature.

Re-running of minimized test example: once you've got a failing test, you can re-run (and debug) it without all the intermediate shrinking steps

[TysonMN]

@moodmosaic, does Haskell Hedgehog have this feature?

[moodmosaic]

@TysonMN no we don't have that yet.

[TysonMN]

Ok, thanks for letting me know.

I will try again this week to implement this feature.

[TysonMN]

This feature is now available in version 0.12.0.

[moodmosaic]

Awesome, great work, @TysonMN.