Performance degradation with 1.4.0

[tmohme]

Using 1.4.0 (as opposed to 1.3.10) we experience a severe performance degradation compared to 1.3.10 for some tests.

The time required to execute (at least some tests) seems to increase super-linear with the number of tries.
According to a first quick glance edgeCases are heavily involved.
I'll try to investigate this further and/or come up with a reproducer in the following days.

One of the affected test classes (containing 17 tests) shows the following behavior:

tries	1.3.10	1.4.0
100	2.5s	4.0s
1000	8s	80s
2000	14s	228s

[jlink]

I've also seen a couple of performance issues today - sadly one day after release :-/

@tmohme Can you share one or two examples that make the problem obvious?

[tmohme]

I found a strong correlation of slowness with usage of custom Arbitraries for heavy domain objects.
I'll try to provide one of these . . .

[jlink]

@tmohme Do you use combinators in you domain arbitraries? That’s what I found at first glance.

[tmohme]

Yes, extensively (at least for these heavy domain objects).

[jlink]

@tmohme That gives me some guidance where to look.

[tmohme]

@jlink Will you be OK with a Kotlin example?
There's lots of code involved and converting it to Java would take some time . . .

[jlink]

@jlink sure

[tmohme]

A small gradle project containing a property-test using an arbitrary-provider for a heavy domain object:
jqwik-166.zip

[jlink]

@tmohme Thanks!

[jlink]

@tmohme Can you try "1.4.1-SNAPSHOT"? It should get rid of the performance degredation; as a matter of fact it should speed up generation compared to 1.3.10 quite a bit. Overall test time of jqwik got down by almost 25%.

Here's the downside: The fix completely breaks the behaviour of Arbitrary.unique() and the @Unique annotation. Both are deprecated anyway but are supposed to stay till 1.5.0. Fixing their behaviour would require a major change in some parts - only to redo everything once they are gone.

It feels like a Catch-22: Either break compatibility despite all the promises I signed with my blood, or leave in the really bad generation performance for composite and flat-map arbitraries, or invest days if not weeks in a deprecated and soon-to-remove functionality.

[osi]

@jlink random thought - what if you "faked" the implementation of how unique works in 1.4?

this is probably super-naive, but it i wonder if may work "well enough" to bridge until the feature's official removal?

[jlink]

@osi That’s actually a good idea. Let’s see what’s possible with limited effort...

[jlink]

@osi Turns out, even a "well enough" implementation requires more effort than I'm willing to invest.

I'll go ahead with a 1.5.0 soonish.

[jlink]

@tmohme I ran your example with 1.3.10 and 10000 tries: 6.9 sec. With 1.5.0-SNAPSHOT: 6.8 sec. Seems 1.5.0-SNAPSHOT should have fixed your problem.

So I'll probably go for a 1.5.0 release tomorrow.

[tmohme]

@jlink I think there is no need to rash this decision to drop backward compatibility so fast:

• I don't know if what we are doing with jqwik is representative or an outlier.
• Everyone who is dissatisfied with 1.4.0 can still use 1.3.10 (as we do for now).
• Also I have no idea how many installations rely on the unique behavior (for us the adaption was a very minor thing).
• Although I guess if your adaptions fix the problem with my example, they will probably also fix our other tests, I can't confirm this for sure until I've tried it (tomorrow).