Test execution much slower compared to Robolectric 1.x

[mpost]

We have quite a few robolectric tests that we migrated from robolectric 1.x to robolectric 2 (2.3-SNAPSHOT). We witnessed that the test execution is much slower compared to robolectric 1.x.

Our 2000+ tests take 3-4 times as long to execute. Eg from 30-40 seconds up to 2 minutes and more. Combined with the gradle startup time the development experience is subpar.

Can we expect to see better robolectric performance in the future?

Is there anything that can be done to speed up the execution? Or any robolectric patterns to avoid?

[JakeWharton]

You can paralellize your JUnit test runner. You can also parallelize Gradle invocation with a CLI flag. Slower tests are the price we pay for using real code instead of the (abysmal) fake code from version 1.

[mpost]

That sounds interesting. Could you point to any specific switches to enable parallel execution?

[JakeWharton]

For Gradle it's just --parallel. JUnit is a bit more complicated. There's a few tutorials you can Google, though.

[mpost]

Ok, I'll experiment with it and report back.

[erd]

Also, keep in mind that you only need to use the Robolectric test runner for tests that touch Android code. You can try and break down your dependencies enough such that you can use mocks to stand in for Android classes and use the normal JUnit test runner.

[mpost]

Sorry for bringing this issue up again but performance of test execution reached an all time low. I am running 2.3-SNAPSHOT on Windows 7 with the gradle-android-test-plugin and have experienced dramatic test execution speed reduction in the last few weeks.

A single robolectric test takes between 0.5 and 1 second. Running a few thousand tests is really becoming a problem when trying to get into a rapid development cycle. Especially since you can not reliably run a single test in AndroidStudio.

The --parallel flag is also not an option as it executes modules in parallel. Not tests themselfs.

[mpost]

It should be mentioned that the performance problems originates from the creation of a new activity in every test. That is a very slow operation.

[xian]

Have you tried running the tests with a profiler to get an idea where the time is spent?

[christopherperry]

I'm seeing the same thing. I tried to mitigate the issue of Robolectric loading my actual Application class by overriding the test runner and always returning a vanilla Application (assuming all my setup code in there was causing the slowdown), but that did basically nothing. Still abysmally slow (489 test in 43s), and I'm on a fast machine.

I just tried parallel execution with the maven surefire plugin, and Robolectric barfs all over the place complaining about things not being on the main thread.

Caused by: java.lang.RuntimeException: you should only be calling this from the main thread!
    at org.robolectric.shadows.ShadowLooper.resetThreadLoopers(ShadowLooper.java:48)
    at org.robolectric.Robolectric.reset(Robolectric.java:1347)
    at org.robolectric.internal.ParallelUniverse.resetStaticState(ParallelUniverse.java:47)
    at org.robolectric.RobolectricTestRunner$2.evaluate(RobolectricTestRunner.java:226)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
    at org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:258)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:744)

[jdmunro]

Does anybody have anything more to add to this topic? I am experiencing similar issues. As @mpost mentioned it takes typically 0.5-1s for creating an activity. For apps with several activities and tests this soon adds up to a huge bottleneck and pretty much stops the ability to TDD activities.

[JakeWharton]

Correctness is more important than speed as we learned from version 1.

If it's causing problems, profile, optimize, and send pull requests. Robolectric is a community project.

[jdmunro]

Thanks for the quick response - contributing is something I will consider. At the moment I just wanted to check whether this was a known bottleneck or perhaps indicative of incorrect usage of the ActivityController, for instance. Also, apologies, I hadn't noticed the "Slower tests are the price we pay for using real code instead of the (abysmal) fake code from version 1." comment above!

With regard to correctness, I fully agree. From a speed perspective, my concern there was that it may be prohibitive to a test-first style approach for developing activities, it is certainly a trade off.

[erd]

For apps with several activities and tests this soon adds up to a huge bottleneck and pretty much stops the ability to TDD activities.

As someone who does TDD all day, every day, I definitely feel your pain. The only suggestion I would add is to see if there is anything you are testing in an activity that you could break out to a separate object and test it outside of an activity (and further from the Android UI framework). Driving an activity through all the necessary lifecycle methods does indeed slow things down a bit.

[jdmunro]

@erd That's a good strategy. Actually, we're just looking into splitting the activity testing into a separate Gradle task altogether which we'll still run, but less often than the regular tests. Though I haven't yet had much luck as my Gradle-fu is still weak.

[mpost]

A strategy that we applied is to reuse the same activity instance throughout many test methods and even test classes. Thereby we speed up our tests quite a bit. Still there are cases where you need to create a new Activity where the speed penalty will hit you.

[jdmunro]

@mpost I was trying something similar actually, do you reuse just the ActivityController or the actual Activity itself? In many of our cases we need the activity to be starting from a "clean" state - how are you handling that?

[mpost]

@jdmunro We are reusing the concrete Activity object. Often times we just need an Activity to instantiate other Views so we would not need to configure the Activity in a special way.

[kingargyle]

@mpost I've gotten around the issue of even need to build the activity by using Mockito to mock it out, and then passing in the mock, or mocking out the findViewById method to return a mock view or inflate it directly. If you can avoid creating an activity for a unit test, do so.

[davcamer]

If you're using roboguice along with robolectric, the container setup can easily be the dominator for test speed. On a project where that was the case, @rcarragher made some changes to roboguice that allowed us to re-use the injector instances from one test to the next. Gave us approximately 30% improvement.

The discussion around it is here:
roboguice/roboguice#178 (comment)

On the same project, we had successfully parallelized the unit test run in maven using the surefire plugin. I think it was parallel at the suite level. I'll try to dig up the config, I need it again for a new project.

[erd]

We've done a few things to help with performance in 2.4. Please re-open if you have more specific things in mind.

[mpost]

While i hope to see a performance gain i just witched from 4.3 to 4.4-SNAPSHOT and i see a HUGHE performance drop. We have around 2500 robolectric tests and they went from 3 minutes to 7-8 minutes.

[ylogx]

Similar problem with 3.0. First test in every suite taking long time. It's like:
class AbcTest:
test1: 18sec
test2: 50ms
test3: 41ms . . .

Is this usual? I don't remember tests taking so much last time I worked with 2.4 tests! We used gradle to run tests back then. I am using junit now.

[bolatug]

My first test on newly created project and I am just trying to robolectric "writing your first test" example. It takes 16 seconds which scares me. Fast feed back is the first reason for TDD. Is there any thing I can do?

org.robolectric:robolectric:3.2.2
targetSdk 24
minSdk 15
my computer is window 8.1 with some Intel i7 processor and 16GB ram

[jongerrish]

That feels very long, compared to robolectrics own tests, which individually run sub-one-second. How big are your unit tests? Do you do a lot of work in your Application.onCreate()? You can profile using YourKit profiler and see where the work is done. Attach your profiler report and if it's robolectric itself we'll take a look and see what can be done.

…

On Jan 12, 2017 11:43 AM, "bolatug" ***@***.***> wrote: My first test on newly created project and I am just trying to robolectric "writing your first test" example. It takes 16 seconds which scares me. Fast feed back is the first reason for TDD. Is there any thing I can do? org.robolectric:robolectric:3.2.2 targetSdk 24 minSdk 15 my computer is window 8.1 with some Intel i7 processor and 16GB ram — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#982 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGBaeAh5YkuGuWf37fB9PPOVg2D7mde3ks5rRoJagaJpZM4BkwB0> .

[bolatug]

I just created the project with an empty activity called MainActivity and just added a Button in to the layout. And created another activity and tried to implement the test case in the site http://robolectric.org/writing-a-test/
So there is only one unit test. And the only logic the project contains is calling activityStart method to open other empty activity.

I couldn't find any resources about how to profile a unit test, can any one provide me some pointers. I am using Android Studio 2.2.3

[mpost]

The instantiation of an Activity takes very very long. Therefore it is advised to reuse an activity throughout multiple tests.