Unable to successfully run macrobenchmark test based off on sample

[spinEdamageD]

I'm trying to run a simple startup test similar to the one shown in the sample here, and although I seem to have everything as shown in the sample gradle file, I keep getting an error when running the test. I'm trying to run directly on a FB device through the gcloud command, using the same command shown in the sample, and although the test seems to start correctly, I keep seeing the following:

java.lang.NoClassDefFoundError: Failed resolution of: Lkotlin/collections/CollectionsKt; at com.sample.macrotests.ColdStartupTest.startup(ColdStartupTest.kt:23) ... 30 trimmed Caused by: java.lang.ClassNotFoundException: kotlin.collections.CollectionsKt ... 32 more

this points to the line where I'm just setting the type of test to run, like:

metrics = listOf(StartupTimingMetric()),

I seem to be missing a dependency but I have everything as shown in the sample so I haven't been able to figure out exactly what I'm missing so not sure if it's a real issue or just something on my side I may be missing so just wanted to call it out here.

Thanks!

[ChrisCraik]

That class is part of the kotlin standard library, which should be bundled into your macrobenchmark apk via the library's dependency.

1. Are you building with gradle, 7.0 or higher?
2. Are you setting experimentalProperties["android.experimental.self-instrumenting"] = true in your macrobenchmark build.gradle as is done here? That's changes the build structure significantly, and allows the test to compile completely separately from the app.
3. Are you using minification in your macrobenchmark module build? That shouldn't be necessary, and won't affect your measurements, since the macrobenchmark builds separately from the app.

[spinEdamageD]

Hey Chris, thanks for the comment.

As for your questions:

1. Yeah, I'm using version 7.0.1
2. I actually have it as properties["android.experimental.self-instrumenting"] = true because if I have it as shown in the docs, I get the following error: Could not get unknown property 'experimentalProperties' for extension 'android' of type com.android.build.gradle.TestExtension. I don't know if this could be an issue but I'm pretty sure I saw that on another working sample so I don't think that's the reason behind the failure.
3. I actually am creating a minimized build for both main and test apk as if I don't add the minimization into both, I get a compilation error complaining about this.

Do you think this might be coming from point 2?

[ChrisCraik]

If it can't find the property, it's likely you need to update your android gradle plugin version. The sample uses 7.0.0-beta04, that or newer should work fine.

[spinEdamageD]

Hey man. Yeah, I've been trying to update that, but for my particular project that seems to be a bit challenging.

Do you know for a fact that it won't work like properties["android.experimental.self-instrumenting"] = true ? I've been trying to find exactly where but I could've sworn I saw it like that in another working sample.

[ChrisCraik]

Without an update to a newer version of the AGP (android gradle plugin), you can make it work, but you won't get the full functionality of the studio / agp integration. You'll also have to have a different module structure temporarily.

For more information, see https://issuetracker.google.com/issues/193896397#comment2

That thread links to an older version of the docs, and the sample, which was built before the AGP support for macrobenchmarks.

[spinEdamageD]

Hey man, thanks again for the quick response. I took a look at the link you mentioned, and it looks like it should work with an older AGP version, because actually in my case I don't even want to run this via AS and I don't really care about trace files but rather I would like to integrate it into a CI flow and I just really care about the single results json file (as of now I just want to be able to trigger the run directly in FB using the gcloud command as seen in the sample), so from the comment in the link, it should work as it is with an older AGP version, so I'm not entirely sure this particular failure is because of this.

I'm still trying to update the AGP version, but I'm out of other ideas as to what to try to make this work.

[ChrisCraik]

If you just care about CI results, you're totally fine going with the com.android.library approach documented in that bug. Basically you have to choose one of the two build structures:

1. com.android.library
   Works on older versions of gradle (back to at least AGP 4.2), documented only in the issue linked above. Note that if you're looking for other examples, you can see usage within the androidx repo (we haven't finished migrating to the new structure).

2. com.android.test (like in this project's sample)
   Must use self-instrumenting property, so requires AGP 7.0.0-beta04 (It may work with older 7.0 betas, not sure which is the first version)

Apologies this isn't more straightforward - our intent was only to support 7.0-beta+, but happy to (best effort) try and help you make it work for older versions, especially if you don't care about local profiling experience.

[spinEdamageD]

Oh thanks a lot man, that looks great, I'll give it a try setting this up as a library module. Also I had another question, in regards to the requirement to have <profileable android:shell="true" /> defined in the manifest, is that something that can be controlled via gradle so basically I can enable it only when I'm building for testing?

The reason I ask is because it seems a bit of a security issue to have that always enabled in the manifest when we only need this for running these tests, so I'd rather not have it always there, and also, is that required if I only care about the single json results file? If this is the case then perhaps I could only add it when I need to build to debug a particular run or trace but leave it out when I'm just running the regular set of tests during CI or local testing.

Thanks again!

[ChrisCraik]

Unfortunately it's only controllable by the manifest currently - filed a bug for that here: https://issuetracker.google.com/194529081

The way that the profileable flag works is it only allows the shell process to profile your app, not any process on the system. Additionally, the main change it performs - enabling android.os.Trace APIs (which is necessary to get useful trace sections from within your process, and make them useful) is actually on by default in Android 12 (1 2). The flag will (eventually) not be required by the library starting with that OS version.

Few things you could try:

1. Use build variants to create a variant of your app with that manifest flag.
2. Enable profileable for all builds. This also enables you to directly profile your release build (with some upcoming changes to the Studio profilers, which will greatly increase accuracy)
3. Suppress the profileable warning and call setAppTracingAllowed from your target app via reflection conditionally. Who needs manifest flags anyway?

[spinEdamageD]

Oh thanks again for the response and for creating the bug.

As for your suggestions, for #1, I do currently have multiple variants, and yeah this is what I ideally want to do, but you mentioned that this is only controlled via the manifest, so how could I do this with build variants?

[ChrisCraik]

I haven't tried it but the docs mention a per-variant or per-build-type source set that gradle uses automatically: https://developer.android.com/studio/build/build-variants#sourceset-build

So in your app, alongside src/main/AndroidManifest.xml you could have src/benchmark/AndroidManifest.xml with just the one profileable tag - that should get enabled only for the benchmark variant by manifest merging (see lower in that page Manifests are merged together into a single manifest. ...).

[spinEdamageD]

Oh got it. Thanks again man. Will give that a try and hopefully that works.

[spinEdamageD]

Hey man, just wanted to let you know that I was able to make it work when I updated according to the docs using AGP v4, and well now that AGP v7 was just released, hopefully soon I can also update that to use the latest docs.

Having said that, I had a couple more questions that I was hoping you could please help me with or at least point me in the right direction.

1. I was curious exactly how the startup macrobenchmark test worked in terms of exactly when it marked the cold start as "complete". I've seen that usually they look at when the main activity is displayed in screen, but if a call to the reportFullyDrawn method was in the code, then that's the point where the cold start was marked as complete. I was hoping that this was also the case for this test, but just wanted to confirm.

2. And secondly, and slightly related to the above. I was wondering if it was possible to set up some sort of 'preparation' code before actually starting the full cold start test. The use case that I want to cover is that I actually need to log in the user before actually being able to properly measure cold start. I've seen that there is a 'setupBlock' in the call to measureRepeated in the benchmark rule, but I'm not sure if that runs in the same process as the test does every time or not, because in the particular cold start scenario I actually need to run this code once, and then afterwards starting the process of the test by killing and restarting the app process and I'm not sure this is how it works.

If you have any insights into the above I would really appreciate your help on this.

Thanks again!

[ChrisCraik]

1. reportFullyDrawn support just landed this week, so you can try it out with a snapshot build now, or wait for the next alpha release, which should contain that change. It's a separate metric reported by StartupTimingMetric, but you'll need to make sure you wait at the end of the test (e.g. using UiAutomator to await for some component to display), so it doesn't complete measurements before your app is done

2. The setupBlock runs after the kill process (see here) that ensures that your startup is cold. You can workaround this by simply calling scope.killProcess again at the end of your setupBlock, but that's isn't especially elegant. Is this login something you'd want to run at the very beginning of a test class? If so, it may make sense to use an @BeforeClass setup step to launch the package yourself, setup login, and then run all of your benchmarks, so you only have to login once. Would be nice to avoid logging in once per benchmark iteration, and setupBlock does just that.