JUnit 5 v/s junit-quickcheck

[felix91gr]

Hi,

First: thank you for implementing junit-quickcheck. I used it this last semester to teach students how to write and use Property Tests, and I would’ve had a much harder time teaching that without this library.

Second: what are the plans for JUnit 5 integration? They seem to be playing with the concept of Dynamic Tests. Although it doesn’t include shrinking afaik and I don’t quite like how the random objects are generated, it seems to be as complete as junit-quickcheck currently is. Am I wrong? Have you guys talked to the JUnit 5 team about this?

Best,
Félix

[pholser]

@felix91gr Thanks for your interest! Glad to hear junit-quickcheck was useful to you and your students.

AFAIK, JUnit 5's "vintage" test engine will be able to run junit-quickcheck tests as-is. Migrating junit-quickcheck to JUnit 5 likely means writing a separate test engine using JUnit 5's new APIs, etc. At some point I'd like to do this, but I'm finding carving out time to work on it difficult.

I know of at least two other viable options in the interim:

• jqwik -- this is a JUnit-5-based test engine that supports property-based tests and example-based tests
• QuickTheories -- this doesn't assume any particular testing framework (JUnit 4, JUnit 5, TestNG, ...)

[pholser]

See also #139.

[pholser]

I don't intend to migrate junit-quickcheck so that it sits atop JUnit 5. Closing this.

[sir4ur0n]

Sorry to reopen this issue, but why don't you want to support JUnit 5?

Use case:
Some features require both unit tests for specific values, and property tests for all other scenarios.

Example:

// Foo.java
Option<Double> inverse(double x) { 
  return x == 0 ? None() : Some(1/x);
}
// FooTest.java
@Test
public void zero() {
  assertThat(foo.inverse(0)).isEmpty();
}

@Property
public void not_zero(double x) {
  assumeThat(x).isNotEqualTo(0);
  assertThat(foo.inverse(x).get() * x).isEqualTo(1); // Don't care about the precision issue here, it's solely for the argument
}

Since we use JUnit 5 everywhere else in the code, and already use some nice features of it (@DisplayName, extensions, etc.), it's a pain in the ass to:

• either have 2 test files for a class (a JUnit 5 for unit tests and a JUnit 4 for property tests)
• or have 1 JUnit 4 test file and we miss many JUnit 5 features (also, double the maintenance on all the other features)

I couldn't find a reason to stick to JUnit 4. Also, since this library is the best PBT library I could find (I considered several, including JUnit 5 dynamic tests and Jqwik/QuickTheories), and judging by the number of github stars, it's the most used, I think it'd make sense to support JUnit 5 which is becoming the testing standard for Java.

The JUnitQuickcheck runner could/should be migrated to an Extension and that would be mostly it in terms of API impacts, don't you think?
Of course, it's easy to say, I don't have (yet?) any idea of the code impact.

[jlink]

As one of the original authors of JUnit 5 and maintainer of a Junit 5 test engine (jqwik.net) I have to say that Jupiter (formerly known as JUnit 5) extensions are probably not sufficient since lifecycle requirements of plain tests and properties differ quite a lot when you look at the details. One might be able to do something like junit-quickcheck with just extensions but it will probably introduce a lot of complications and strange behaviour when combined with other extensions. That's why others, e.g. jqwik, use JUnit 5 test engines for similar purposes. Test engines have full control of the lifecycle but require more implementation.

BTW, in theory it should be possible to mix Jupiter tests and JUnit 4 (called vintage) tests in the same class - and thereby also junit-quickcheck properties. I haven't tried it, though, and you must be careful not to mix up the @Test annotation which exists twice in different packages.

[sir4ur0n]

I may have missed something, but I took several looks and I never saw a way to use both JUnit 4 (and a specific runner) and JUnit 5 (and extensions) in the same file. If such a thing is possible, I would love to get a link or example to work it out for my needs from there.

Slightly digressing, I checked Jqwik a few times but there were several things that bugged me (maybe for wrong reasons or I misunderstood):

• Cannot reuse extensions (e.g. Mockito one).
• "Reinventing the wheel": a lot of features already exist in Jupiter and I feel like everybody would be better off if a sub-library jupiter-property were integrated into Jupiter rather than maintaining yet another engine. Also, this repels a lot of contributors/users as it doesn't benefit JUnit popularity
• Spamming @Forall left and right 😞

[jlink]

Here are two examples to combine Jupiter with Vintage/junit-quickcheck:

https://github.com/jlink/property-based-testing/blob/master/pbt-java/src/test/java/junitquickcheck/JupiterAndVintageTests.java

https://github.com/jlink/property-based-testing/blob/master/pbt-java/src/test/java/junitquickcheck/JupiterAndQuickcheckTests.java

The lifecycles of Vintage and Jupiter tests are separated, i.e., two instances of the testclass are being created. And, obviously, Jupiter extensions still do not work with junit-quickcheck.

As for the jqwik issues and questions: Take them over to https://github.com/jlink/jqwik and I'll be more than happy to answer.