Jetpack Macrobenchmark takes a too long time to complete the test, its stuck in 97%

[MML-Pro]

IDE info

Android Studio Dolphin | 2021.3.1
Build #AI-213.7172.25.2113.9014738, built on August 31, 2022
Runtime version: 11.0.13+0-b1751.21-8125866 amd64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o.
Windows 10 10.0
GC: G1 Young Generation, G1 Old Generation
Memory: 2024M
Cores: 8
Registry:
    external.system.auto.import.disabled=true
    ide.text.editor.with.preview.show.floating.toolbar=false

Non-Bundled Plugins:
    wu.seal.tool.jsontokotlin (3.7.4)
    com.thoughtworks.gauge (213.5744.125)
    drawable.importer.pluginId.7658.hack' (0.8)
    de.ohmesoftware.parcelablegenerator (0.7.1)

I tried to test this sample app on my android device and emulator, it's taking a too long time (50 mins till now) to complete the test, it seems stuck on 97%

Starting a Gradle Daemon, 2 incompatible and 3 stopped Daemons could not be reused, use --status for details

Task :macrobenchmark:connectedBenchmarkAndroidTest
Starting 34 tests on MI 8 Lite - 11
<============-> 97% EXECUTING [50m 39s]
IDLE
IDLE
IDLE
:macrobenchmark:connectedBenchmarkAndroidTest

I've also created the issue here

[keyboardsurfer]

Thanks for reporting this problem.
Is this an issue with the currently existing sample from main?

[MML-Pro]

Thanks for reporting this problem. Is this an issue with the currently existing sample from main?

If I understand your question right this happens in the MacrobenchmarkSample sample app, I didn't test/tried the other samples JankStats sample or Microbenchmark sample

[keyboardsurfer]

I can't reproduce this behavior on either a Mac nor a Linux machine and currently don't have access to a Windows machine.
I'm not aware of any general issues with Windows machines and Macrobenchmarking.
Could you check your build outputs and gradle logs for more detailed logs?

[MML-Pro]

Hello Ben,
I tried to restart my device and run it again on new version of android and I got this

terminal output

Windows PowerShell                                          
Copyright (C) Microsoft Corporation. All rights reserved.   
                                                            
Try the new cross-platform PowerShell https://aka.ms/pscore6

PS C:\Users\Mido\StudioProjects\performance-samples\MacrobenchmarkSample> ./gradlew macrobenchmark:cC
Starting a Gradle Daemon, 1 busy and 1 incompatible Daemons could not be reused, use --status for details

> Task :macrobenchmark:connectedBenchmarkAndroidTest
Starting 34 tests on MI 8 Lite - 12

com.example.macrobenchmark.TextInputFrameTimingBenchmark > inputText[MI 8 Lite - 12] FAILED 
        java.util.NoSuchElementException: No element of the collection was transformed to a non-null value.
        at androidx.benchmark.macro.FrameStatsResult$Companion.parse(FrameStatsResult.kt:57)

com.example.macrobenchmark.TrivialBaselineProfileBenchmark > appStartupAndUserJourneys[MI 8 Lite - 12] FAILED 
        java.lang.IllegalArgumentException: Baseline Profile Collection requires a rooted device, and a rooted adb session. Use `adb root`.
        at androidx.benchmark.macro.BaselineProfilesKt.collectBaselineProfile(BaselineProfiles.kt:45)

com.example.macrobenchmark.TrivialBaselineProfileBenchmark > appStartupOnly[MI 8 Lite - 12] FAILED 
        java.lang.IllegalArgumentException: Baseline Profile Collection requires a rooted device, and a rooted adb session. Use `adb root`.
        at androidx.benchmark.macro.BaselineProfilesKt.collectBaselineProfile(BaselineProfiles.kt:45)

com.example.macrobenchmark.clicks.ClickLatencyBenchmark > recyclerViewClick[MI 8 Lite - 12] FAILED 
        java.util.NoSuchElementException: No element of the collection was transformed to a non-null value.
        at androidx.benchmark.macro.FrameStatsResult$Companion.parse(FrameStatsResult.kt:57)

com.example.macrobenchmark.clicks.ClickLatencyBenchmark > simpleViewClick[MI 8 Lite - 12] FAILED 
        java.util.NoSuchElementException: No element of the collection was transformed to a non-null value.
        at androidx.benchmark.macro.FrameStatsResult$Companion.parse(FrameStatsResult.kt:57)

com.example.macrobenchmark.clicks.ClickLatencyBenchmark > scrollViewClick[MI 8 Lite - 12] FAILED 
        java.util.NoSuchElementException: No element of the collection was transformed to a non-null value.
        at androidx.benchmark.macro.FrameStatsResult$Companion.parse(FrameStatsResult.kt:57)

com.example.macrobenchmark.clicks.ClickLatencyBenchmark > composeLazyColumnClick[MI 8 Lite - 12] FAILED 
        java.util.NoSuchElementException: No element of the collection was transformed to a non-null value.
        at androidx.benchmark.macro.FrameStatsResult$Companion.parse(FrameStatsResult.kt:57)

com.example.macrobenchmark.clicks.ClickLatencyBenchmark > listViewClick[MI 8 Lite - 12] FAILED 
        java.util.NoSuchElementException: No element of the collection was transformed to a non-null value.
        at androidx.benchmark.macro.FrameStatsResult$Companion.parse(FrameStatsResult.kt:57)

com.example.macrobenchmark.frames.FrameTimingBenchmark > scrollList[MI 8 Lite - 12] FAILED 
        java.util.NoSuchElementException: No element of the collection was transformed to a non-null value.
        at androidx.benchmark.macro.FrameStatsResult$Companion.parse(FrameStatsResult.kt:57)

com.example.macrobenchmark.frames.FrameTimingBenchmark > scrollComposeList[MI 8 Lite - 12] FAILED 
        java.util.NoSuchElementException: No element of the collection was transformed to a non-null value.
        at androidx.benchmark.macro.FrameStatsResult$Companion.parse(FrameStatsResult.kt:57)

com.example.macrobenchmark.frames.NestedRecyclerFrameTimingBenchmarks > scrollNestedRecyclerWithRecyclerPool[MI 8 Lite - 12] FAILED 
        java.util.NoSuchElementException: No element of the collection was transformed to a non-null value.
        at androidx.benchmark.macro.FrameStatsResult$Companion.parse(FrameStatsResult.kt:57)

com.example.macrobenchmark.frames.NestedRecyclerFrameTimingBenchmarks > scrollNestedRecyclerWithoutRecyclerPool[MI 8 Lite - 12] FAILED 
        java.util.NoSuchElementException: No element of the collection was transformed to a non-null value.
        at androidx.benchmark.macro.FrameStatsResult$Companion.parse(FrameStatsResult.kt:57)

com.example.macrobenchmark.frames.NonExportedActivityBenchmark > nonExportedActivityScrollList[MI 8 Lite - 12] FAILED 
        java.util.NoSuchElementException: No element of the collection was transformed to a non-null value.
        at androidx.benchmark.macro.FrameStatsResult$Companion.parse(FrameStatsResult.kt:57)

com.example.macrobenchmark.startup.ColdStartupBenchmark > startupPartialWithBaselineProfiles[MI 8 Lite - 12] FAILED 
        java.lang.RuntimeException: Baseline profiles aren't supported on this device version
        at androidx.benchmark.macro.CompilationMode$Partial.broadcastBaselineProfileInstall(CompilationMode.kt:307)

com.example.macrobenchmark.startup.ColdStartupBenchmark > startupNoCompilation[MI 8 Lite - 12] FAILED 
        java.util.NoSuchElementException: No element of the collection was transformed to a non-null value.
        at androidx.benchmark.macro.FrameStatsResult$Companion.parse(FrameStatsResult.kt:57)

com.example.macrobenchmark.startup.ColdStartupBenchmark > startupFullCompilation[MI 8 Lite - 12] FAILED 
        java.util.NoSuchElementException: No element of the collection was transformed to a non-null value.
        at androidx.benchmark.macro.FrameStatsResult$Companion.parse(FrameStatsResult.kt:57)

com.example.macrobenchmark.startup.ColdStartupBenchmark > startupPartialCompilation[MI 8 Lite - 12] FAILED 
        java.util.NoSuchElementException: No element of the collection was transformed to a non-null value.
        at androidx.benchmark.macro.FrameStatsResult$Companion.parse(FrameStatsResult.kt:57)

com.example.macrobenchmark.startup.FullyDrawnStartupBenchmark > startup[MI 8 Lite - 12] FAILED 

Tests on MI 8 Lite - 12 failed: There was 18 failure(s).
Test run failed to complete. Expected 34 tests, received 17

> Task :macrobenchmark:connectedBenchmarkAndroidTest FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':macrobenchmark:connectedBenchmarkAndroidTest'.
> There were failing tests. See the report at: file:///C:/Users/Mido/StudioProjects/performance-samples/MacrobenchmarkSample/macrobenchmark/build/reports/androidTe
sts/connected/index.html

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 5m 30s
87 actionable tasks: 2 executed, 85 up-to-date
PS C:\Users\Mido\StudioProjects\performance-samples\MacrobenchmarkSample> 

[keyboardsurfer]

Baseline Profile Collection requires a rooted device, and a rooted adb session. Use adb root

Please follow this instruction from the log and let me know how the situation improves.

[MML-Pro]

Baseline Profile Collection requires a rooted device, and a rooted adb session. Use adb root

Please follow this instruction from the log and let me know how the situation improves.

Sorry I didn't understand your last comment, my device already rooted, what should I do before run this command ./gradlew macrobenchmark:cC ?

[keyboardsurfer]

adb root , then run the benchmark.

[mrzbn]

@keyboardsurfer

my app's locale is Farsi and I encounter this exception, too.

java.util.NoSuchElementException: No element of the collection was transformed to a non-null value. at androidx.benchmark.macro.FrameStatsResult$Companion.parse(FrameStatsResult.kt:57)

I checked and realized that parse method looks for following regex in the output:

private val NAME_REGEX = Regex("(\\S+) \\(visibility=[0-9]+\\)")

and my output contains: (visibility=۰)

"۰" is 0 in Farsi. I think that's the reason parsing throws exception.

[keyboardsurfer]

Thanks for the context, @mrzbn. I have filed a bug to keep track of this problem.

[YingChen0605]

FYI When I try running benchmark in a rooted Pixel 7 pro, it takes forever to start up the test. But after I unroot the device, the test can complete successfully very quickly.

[keyboardsurfer]

This has been fixed in AOSP and will be published with an upcoming release.