Arbitraries containing null don't work correctly when used exhaustively

[rkraneis]

We want to exhaustively test parameters that can be either null or zero. This seems to be not possible at the moment. In the cases where jqwik automatically selects exhaustive generation (nullOrZero 1, 2 and 4), only null or zero is ever generated, depending on how the arbitrary is created. Forcing random generation on the other hand leads to the generation of both null and zero.

public class NullOrZeroTest {

  @Property @Report(Reporting.GENERATED)
  boolean nullOrZero1(@ForAll("nullOrZero1") BigDecimal bd) {
    System.out.println("nullOrZero1 " + bd);
    return bd == null || bd.compareTo(BigDecimal.ZERO) == 0;
  }

  @Property @Report(Reporting.GENERATED)
  boolean nullOrZero2(@ForAll("nullOrZero2") BigDecimal bd) {
    System.out.println("nullOrZero2 " + bd);
    return bd == null || bd.compareTo(BigDecimal.ZERO) == 0;
  }

  @Property(generation = GenerationMode.EXHAUSTIVE) @Report(Reporting.GENERATED)
  boolean nullOrZero3(@ForAll("nullOrZero3") BigDecimal bd) {
    System.out.println("nullOrZero3 " + bd);
    return bd == null || bd.compareTo(BigDecimal.ZERO) == 0;
  }

  @Property @Report(Reporting.GENERATED)
  boolean nullOrZero4(@ForAll("nullOrZero4") BigDecimal bd) {
    System.out.println("nullOrZero4 " + bd);
    return bd == null || bd.compareTo(BigDecimal.ZERO) == 0;
  }

  @Provide Arbitrary<BigDecimal> nullOrZero1() {
    return Arbitraries.of(null, BigDecimal.ZERO);
  }

  @Provide Arbitrary<BigDecimal> nullOrZero2() {
    return Arbitraries.of(BigDecimal.ZERO, null);
  }

  @Provide Arbitrary<BigDecimal> nullOrZero3() {
    return Arbitraries.bigDecimals().between(BigDecimal.ZERO, BigDecimal.ZERO)
        .withSamples((BigDecimal) null);
  }

  @Provide Arbitrary<BigDecimal> nullOrZero4() {
    return Arbitraries.of(new BigDecimal("0"), new BigDecimal("0.0"), new BigDecimal("0.00"))
        .withSamples((BigDecimal) null);
  }
}

Test output:

Running NullOrZeroTest
timestamp = 2019-12-12T20:26:00.925608, generated = [null]

nullOrZero1 null
timestamp = 2019-12-12T20:26:00.944175, generated = [0]

nullOrZero2 0
timestamp = 2019-12-12T20:26:00.944756, generated = [null]

nullOrZero2 null
timestamp = 2019-12-12T20:26:00.951872, NullOrZeroTest:nullOrZero2 = 
                              |-------------------jqwik-------------------
tries = 2                     | # of calls to property
checks = 2                    | # of not rejected calls
generation-mode = EXHAUSTIVE  | parameters are exhaustively generated
after-failure = PREVIOUS_SEED | use the previous seed
seed = 4566159385613353648    | random seed to reproduce generated values


timestamp = 2019-12-12T20:26:00.973788, generated = [null]

nullOrZero4 null
Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.152 s <<< FAILURE! - in NullOrZeroTest
nullOrZero3  Time elapsed: 0.015 s  <<< ERROR!
net.jqwik.api.JqwikException: EXHAUSTIVE generation is not possible. Maybe too many potential examples?

╷
├─ jqwik (JUnit Platform) ✔
│  └─ NullOrZeroTest ✔
│     ├─ nullOrZero1 ✔
│     ├─ nullOrZero2 ✔
│     ├─ nullOrZero3 ✘ EXHAUSTIVE generation is not possible. Maybe too many potential examples?
│     └─ nullOrZero4 ✔
└─ JUnit Jupiter ✔

Results:

Errors: 
  NullOrZeroTest.nullOrZero3 » Jqwik EXHAUSTIVE generation is not possible. Mayb...

Tests run: 4, Failures: 0, Errors: 1, Skipped: 0

Edit: The nullOrZero2 combination actually does work, but the others not. So it seems null must not be the first value created.
Edit2: Even then we do not get the full test matrix when testing two parameters:

Running NullOrZeroTest
timestamp = 2019-12-12T20:49:32.218276, generated = [0, 0]

timestamp = 2019-12-12T20:49:32.220822, generated = [0, null]

timestamp = 2019-12-12T20:49:32.221372, generated = [null, 0]

timestamp = 2019-12-12T20:49:32.231031, NullOrZeroTest:nullOrZero2 = 
                              |-------------------jqwik-------------------
tries = 3                     | # of calls to property
checks = 3                    | # of not rejected calls
generation-mode = EXHAUSTIVE  | parameters are exhaustively generated
after-failure = PREVIOUS_SEED | use the previous seed
seed = 8110854518611637746    | random seed to reproduce generated values

[rkraneis]

Using @ForAll BigRange(min="0",max="0") WithNull(0.5) BigDecimal bd seems equivalent to nullOrZero3 and cannot be done exhaustively.

[rkraneis]

Minimal test for failing combination:

public class NullOrZeroTest {
  
  Object ZERO = new Object();

  @Property @Report(Reporting.GENERATED)
  boolean nullOrZero2(@ForAll("nullOrZero2") Object bd1, @ForAll("nullOrZero2") Object bd2) {
    return bd1 == null || bd1 == ZERO || bd2 == null || bd2 == ZERO;
  }

  @Provide Arbitrary<Object> nullOrZero2() {
    return Arbitraries.of(ZERO, null);
  }
}

[jlink]

I'm just starting to try understanding what you want to do so bear with me...

Your last example NullOrZeroTest succeeds when I try it:

timestamp = 2019-12-13T10:46:34.003, generated = [java.lang.Object@1e9e725a, java.lang.Object@1e9e725a]

timestamp = 2019-12-13T10:46:34.006, generated = [java.lang.Object@1e9e725a, null]

timestamp = 2019-12-13T10:46:34.006, generated = [null, java.lang.Object@1e9e725a]

timestamp = 2019-12-13T10:46:34.016, NullOrZeroTest:nullOrZero2 = 
                              |-------------------jqwik-------------------
tries = 3                     | # of calls to property
checks = 3                    | # of not rejected calls
generation-mode = EXHAUSTIVE  | parameters are exhaustively generated
after-failure = PREVIOUS_SEED | use the previous seed
seed = -5796249568264761713   | random seed to reproduce generated values

This is almost what I'd expect. I'm just wondering why [null, null] does not show up as 4th example and I will look into this. Is that your point?

When I switch on randomized generation with @Property(generation = GenerationMode.RANDOMIZED) it does show up, though.

[jlink]

@Provide Arbitrary<BigDecimal> nullOrZero3() {
    return Arbitraries.bigDecimals().between(BigDecimal.ZERO, BigDecimal.ZERO)
        .withSamples((BigDecimal) null);
  }

should also be able to be used in an exhaustive generation context. I'll have a look at that as well.

[rkraneis]

Ah yes, sorry, that might have been too much information with too little explanation ;-)
Given Arbitraries.of(null, ZERO)

• it creates null as the only value when used in exhaustive mode for a single parameter

Given Arbitraries.of(ZERO, null)

• it creates ZERO and null as values when used in exhaustive mode for a single parameter
• it creates all permutations but [null, null] when used in exhaustive mode for two parameters

Randomized generation is fine in all cases, but must be selected manually.

[jlink]

Understood. I'm going to examine that.

[jlink]

Bug concerning handling of null values in exhaustive generation has been fixed. Already available in version 1.2.2-SNAPSHOT. See 24ff64c

Exhaustive generation of Arbitraries.bigDecimals().between(BigDecimal.ZERO, BigDecimal.ZERO) is a different beast, which I will tackle next.

[rkraneis]

Thank you for the quick response!

[jlink]

The exhaustive generation of single big decimal range is now also possible (1e4cc24) and available in 1.2.2-SNAPSHOT

All of your examples above should now hopefully work as expected.

Thanks for catching two glitches in a single go!

[rkraneis]

Great, thank you!