Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Arbitraries containing null don't work correctly when used exhaustively #77

Closed
rkraneis opened this issue Dec 12, 2019 · 10 comments
Closed

Arbitraries containing null don't work correctly when used exhaustively #77

rkraneis opened this issue Dec 12, 2019 · 10 comments
Labels
bug

Comments

@rkraneis
Copy link

@rkraneis rkraneis commented Dec 12, 2019

We want to exhaustively test parameters that can be either null or zero. This seems to be not possible at the moment. In the cases where jqwik automatically selects exhaustive generation (nullOrZero 1, 2 and 4), only null or zero is ever generated, depending on how the arbitrary is created. Forcing random generation on the other hand leads to the generation of both null and zero.

public class NullOrZeroTest {

  @Property @Report(Reporting.GENERATED)
  boolean nullOrZero1(@ForAll("nullOrZero1") BigDecimal bd) {
    System.out.println("nullOrZero1 " + bd);
    return bd == null || bd.compareTo(BigDecimal.ZERO) == 0;
  }

  @Property @Report(Reporting.GENERATED)
  boolean nullOrZero2(@ForAll("nullOrZero2") BigDecimal bd) {
    System.out.println("nullOrZero2 " + bd);
    return bd == null || bd.compareTo(BigDecimal.ZERO) == 0;
  }

  @Property(generation = GenerationMode.EXHAUSTIVE) @Report(Reporting.GENERATED)
  boolean nullOrZero3(@ForAll("nullOrZero3") BigDecimal bd) {
    System.out.println("nullOrZero3 " + bd);
    return bd == null || bd.compareTo(BigDecimal.ZERO) == 0;
  }

  @Property @Report(Reporting.GENERATED)
  boolean nullOrZero4(@ForAll("nullOrZero4") BigDecimal bd) {
    System.out.println("nullOrZero4 " + bd);
    return bd == null || bd.compareTo(BigDecimal.ZERO) == 0;
  }

  @Provide Arbitrary<BigDecimal> nullOrZero1() {
    return Arbitraries.of(null, BigDecimal.ZERO);
  }

  @Provide Arbitrary<BigDecimal> nullOrZero2() {
    return Arbitraries.of(BigDecimal.ZERO, null);
  }

  @Provide Arbitrary<BigDecimal> nullOrZero3() {
    return Arbitraries.bigDecimals().between(BigDecimal.ZERO, BigDecimal.ZERO)
        .withSamples((BigDecimal) null);
  }

  @Provide Arbitrary<BigDecimal> nullOrZero4() {
    return Arbitraries.of(new BigDecimal("0"), new BigDecimal("0.0"), new BigDecimal("0.00"))
        .withSamples((BigDecimal) null);
  }
}

Test output:

Running NullOrZeroTest
timestamp = 2019-12-12T20:26:00.925608, generated = [null]

nullOrZero1 null
timestamp = 2019-12-12T20:26:00.944175, generated = [0]

nullOrZero2 0
timestamp = 2019-12-12T20:26:00.944756, generated = [null]

nullOrZero2 null
timestamp = 2019-12-12T20:26:00.951872, NullOrZeroTest:nullOrZero2 = 
                              |-------------------jqwik-------------------
tries = 2                     | # of calls to property
checks = 2                    | # of not rejected calls
generation-mode = EXHAUSTIVE  | parameters are exhaustively generated
after-failure = PREVIOUS_SEED | use the previous seed
seed = 4566159385613353648    | random seed to reproduce generated values


timestamp = 2019-12-12T20:26:00.973788, generated = [null]

nullOrZero4 null
Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.152 s <<< FAILURE! - in NullOrZeroTest
nullOrZero3  Time elapsed: 0.015 s  <<< ERROR!
net.jqwik.api.JqwikException: EXHAUSTIVE generation is not possible. Maybe too many potential examples?

╷
├─ jqwik (JUnit Platform) ✔
│  └─ NullOrZeroTest ✔
│     ├─ nullOrZero1 ✔
│     ├─ nullOrZero2 ✔
│     ├─ nullOrZero3 ✘ EXHAUSTIVE generation is not possible. Maybe too many potential examples?
│     └─ nullOrZero4 ✔
└─ JUnit Jupiter ✔

Results:

Errors: 
  NullOrZeroTest.nullOrZero3 » Jqwik EXHAUSTIVE generation is not possible. Mayb...

Tests run: 4, Failures: 0, Errors: 1, Skipped: 0

Edit: The nullOrZero2 combination actually does work, but the others not. So it seems null must not be the first value created.
Edit2: Even then we do not get the full test matrix when testing two parameters:

Running NullOrZeroTest
timestamp = 2019-12-12T20:49:32.218276, generated = [0, 0]

timestamp = 2019-12-12T20:49:32.220822, generated = [0, null]

timestamp = 2019-12-12T20:49:32.221372, generated = [null, 0]

timestamp = 2019-12-12T20:49:32.231031, NullOrZeroTest:nullOrZero2 = 
                              |-------------------jqwik-------------------
tries = 3                     | # of calls to property
checks = 3                    | # of not rejected calls
generation-mode = EXHAUSTIVE  | parameters are exhaustively generated
after-failure = PREVIOUS_SEED | use the previous seed
seed = 8110854518611637746    | random seed to reproduce generated values
@rkraneis rkraneis changed the title Arbitraries containing null don't work correctly Arbitraries containing null don't work correctly when used exhaustively Dec 12, 2019
@rkraneis

This comment has been minimized.

Copy link
Author

@rkraneis rkraneis commented Dec 12, 2019

Using @ForAll BigRange(min="0",max="0") WithNull(0.5) BigDecimal bd seems equivalent to nullOrZero3 and cannot be done exhaustively.

@rkraneis

This comment has been minimized.

Copy link
Author

@rkraneis rkraneis commented Dec 12, 2019

Minimal test for failing combination:

public class NullOrZeroTest {
  
  Object ZERO = new Object();

  @Property @Report(Reporting.GENERATED)
  boolean nullOrZero2(@ForAll("nullOrZero2") Object bd1, @ForAll("nullOrZero2") Object bd2) {
    return bd1 == null || bd1 == ZERO || bd2 == null || bd2 == ZERO;
  }

  @Provide Arbitrary<Object> nullOrZero2() {
    return Arbitraries.of(ZERO, null);
  }
}
@jlink

This comment has been minimized.

Copy link
Owner

@jlink jlink commented Dec 13, 2019

I'm just starting to try understanding what you want to do so bear with me...

Your last example NullOrZeroTest succeeds when I try it:

timestamp = 2019-12-13T10:46:34.003, generated = [java.lang.Object@1e9e725a, java.lang.Object@1e9e725a]

timestamp = 2019-12-13T10:46:34.006, generated = [java.lang.Object@1e9e725a, null]

timestamp = 2019-12-13T10:46:34.006, generated = [null, java.lang.Object@1e9e725a]

timestamp = 2019-12-13T10:46:34.016, NullOrZeroTest:nullOrZero2 = 
                              |-------------------jqwik-------------------
tries = 3                     | # of calls to property
checks = 3                    | # of not rejected calls
generation-mode = EXHAUSTIVE  | parameters are exhaustively generated
after-failure = PREVIOUS_SEED | use the previous seed
seed = -5796249568264761713   | random seed to reproduce generated values

This is almost what I'd expect. I'm just wondering why [null, null] does not show up as 4th example and I will look into this. Is that your point?

When I switch on randomized generation with @Property(generation = GenerationMode.RANDOMIZED) it does show up, though.

@jlink jlink added the bug label Dec 13, 2019
@jlink

This comment has been minimized.

Copy link
Owner

@jlink jlink commented Dec 13, 2019

@Provide Arbitrary<BigDecimal> nullOrZero3() {
    return Arbitraries.bigDecimals().between(BigDecimal.ZERO, BigDecimal.ZERO)
        .withSamples((BigDecimal) null);
  }

should also be able to be used in an exhaustive generation context. I'll have a look at that as well.

@rkraneis

This comment has been minimized.

Copy link
Author

@rkraneis rkraneis commented Dec 13, 2019

Ah yes, sorry, that might have been too much information with too little explanation ;-)
Given Arbitraries.of(null, ZERO)

  • it creates null as the only value when used in exhaustive mode for a single parameter

Given Arbitraries.of(ZERO, null)

  • it creates ZERO and null as values when used in exhaustive mode for a single parameter
  • it creates all permutations but [null, null] when used in exhaustive mode for two parameters

Randomized generation is fine in all cases, but must be selected manually.

@jlink

This comment has been minimized.

Copy link
Owner

@jlink jlink commented Dec 13, 2019

Understood. I'm going to examine that.

@jlink

This comment has been minimized.

Copy link
Owner

@jlink jlink commented Dec 13, 2019

Bug concerning handling of null values in exhaustive generation has been fixed. Already available in version 1.2.2-SNAPSHOT. See 24ff64c

Exhaustive generation of Arbitraries.bigDecimals().between(BigDecimal.ZERO, BigDecimal.ZERO) is a different beast, which I will tackle next.

@rkraneis

This comment has been minimized.

Copy link
Author

@rkraneis rkraneis commented Dec 13, 2019

Thank you for the quick response!

@jlink

This comment has been minimized.

Copy link
Owner

@jlink jlink commented Dec 13, 2019

The exhaustive generation of single big decimal range is now also possible (1e4cc24) and available in 1.2.2-SNAPSHOT

All of your examples above should now hopefully work as expected.

Thanks for catching two glitches in a single go!

@jlink jlink closed this Dec 13, 2019
@rkraneis

This comment has been minimized.

Copy link
Author

@rkraneis rkraneis commented Dec 16, 2019

Great, thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.