Performance degradation after migration to mockito 2

[kajf]

Hey,

I am facing severe test suite execution performance degradation during Mockito migration
from version 1.10.19 to 2.28.2

Whole migrated test suite (about 12k tests) run takes ~18 seconds while after migration it took around 2 minutes. I have checked previous performance related tickets, but those are supposed to be resolved in version I use for migration.

Initial setup
java version: 1.8.0_112
junit version: 4.12
mockito version: 1.10.19

Migrated setup
java version: 1.8.0_112
junit version: 4.12
mockito version: 2.28.2

30,000 mocks
Test Results (in milliseconds) on my laptop (8 Core, 16 Gb RAM)

run	mockito-one	mockito-two
1	1,847	2,485
2	1,984	3,118
3	2,039	3,716
4	2,115	2,942
5	1,963	2,498

It seems to be related to the way method mocking is done, since blank mock creation shows similar results for both mockito versions.

See two small test projects (for mockito verstion 1 and version 2) attached
perf_test.zip

[TimvdLippe]

Could you do a performance trace with https://visualvm.github.io/ and analyze where we lose the most amount of time?

[TimvdLippe]

And it is not totally unexpected the time has increased, as the ByteBuddyMockmaker actually follow the Java specification, which the CGLibMockmaker did not. However, such a large performance regression is unexpected.

[kajf]

It looks like most of the time (about 1 minute) is spent in org.mockito.internal.runners.DefaultInternalRunner$1$1.evaluate()
which did not seem to be called in mockito 1 during org.junit.rules.RunRules.evaluate()
See snapshot attached
slow-mock.zip

[TimvdLippe]

That makes sense, as that is the overall wrapping runner. I guess we need a little bit more fine-grained data. E.g. if you ignore the infrastructure related to JUnit, what part of the Mockito internal system is most used.

Maybe to ease the debugging, could you post a screenshot of the flamechart of the methods that are most often called?

[kajf]

Hope this helps

See attached full profiler result
mock-flame.zip

[TimvdLippe]

Thank you for the flamechart. The work in ByteBuddy definitely introduces a large amount of overhead for the simple test. However, this is not totally unexpected for the creation of 30000 mocks.

Assigning @raphw for now who might have some more thoughts on how to speed up ByteBuddy, but in general I think trying to limit the amount of mocks you create in your test suite is a better solution.

Do you have any numbers on how many mocks are getting created in your own test suite?

[raphw]

Mock creation is more expensive in Byte Buddy compared to cglib since Byte Buddy processes generic data to get bridges right. This alone takes a lot of additional time. Normally, this is not sn issue since nobody would mock 30k classes in a two second test. The Byte Buddy based mocks are more performant, as a matter of fact and would amortize their costs in a long running applications but this rarely plays a role in test suites, especially not in one like yours.

Are you sure this is the performance isdue you are experiencing? Does this also degrade performance of an actual test suite? There might be other reasons that your actual tests became slower. If you mock 200 classes that should not take more than 20 milliseconds even on a cold VM. Unless you are doing something particular, this change should not normally impact you.

[kajf]

@TimvdLippe
The flamechart and VisualVM snapshot I provided was for the real suite I am migrating, not for the artificial run provided in issue description.

Real suite I am trying to migrate to Mockito 2 have the following stats
number of mocks with Mockito.mock() ~ 4400
number of mocks with @mock ~ 3200
number of spies with Mockito.spy() ~ 700
number of spies with @SPY ~ 90
number of tests ~ 11800

runtime with Mockito 1 ~ 17 sec
runtime with Mockito 2 ~ 1 min 45 sec

So in total it is a ~ 9k of mocks causing degradation ~5 times

Perhaps the provided test example does not highlight the whole issue, but hopefully the profiler snapshots do, since runtime of unit tests becomes a blocker for migration

Let me know if I can provide anything else to help resolve the issue

[TimvdLippe]

I think that is the trade-off we accepted. We opted for a more correct Mockmaker in exchange for a small loss of performance on initial mock creation (which the CGLibMockmaker did). However, if your test has a lot of mock creation without a lot of mock interaction, this cost will trump any other cost in your unit test.

At this point, I think the best way forward is to limit the amount of mocks you create in your unit tests. Creating 9000 mocks is a lot and I don't think any times of performance improvements on the side of ByteBuddy is ever going to offset the amount of pressure being put on the dynamic class generation.

If you are still able to reproduce this issue with way fewer mocks, then there is definitely a performance regression we should fix. But currently I would say this is WAI and expected to be more computationally expensive, given the amount of mocks.

[ramSilva]

I see a huge performance increase (25ms vs 430ms in a very simple test case with just one mock) when instantiating mocks inline val mock = mock(SomeClass) vs when using an annotation @Mock val mock: SomeClass / instantiating in the setup method.

Is this expected or is it wrong to create mocks inline?

Some further details here https://stackoverflow.com/questions/59390997/why-do-i-see-a-performance-boost-when-creating-mockito-mocks-inline

[raphw]

Might this be a Kotlin-related thing? This should really not be the case if you instantiate the same amount of mocks. Mockito - like any Java program - is not aware of the caller of its methods, therefore it should not really make a difference.

Could you run both scenarios with a profiler attached and see where the additional time is spent relatively?

[ramSilva]

Well, turns out the "performance increase" was just android studio being misleading with test execution times. I used JProfiler and saw that both inlining the method call and doing it in the setup method leads to a call to Mockito.mock() that takes the same time to execute.

The only difference is where in the stack trace said call is performed, so I'm assuming android studio only starts counting after a certain point, which leads to the very low execution times.

[ShriprasadM]

Can we use mockito 3.2.4 instead (https://search.maven.org/artifact/org.mockito/mockito-core)? Are there any performan ce issues?

[kajf]

sorry to revive this old thread
I have tried the migration from 1.10.19 this time to 3.7.7 version of Mockito
unfortunately with similar performance results

13.4k tests takes more than 2 minutes vs 22 seconds with old Mockito version