Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Execute Additional Test apks #511

Open
inktomi opened this Issue Mar 1, 2019 · 13 comments

Comments

4 participants
@inktomi
Copy link

inktomi commented Mar 1, 2019

For projects that have many modules it's not uncommon to include some small number of android tests in a submodule. Flank is currently unable to run these tests because they are not packaged in the main app's test apk.

In order to run these tests today, a user would need to write a script that passes each library to flank along with the main app's apk for the "app". This is doable, but it would be better if flank could handle this for us.

I'd like to propose the following:

  1. A new --module-apks parameter in the Flank yaml. This is a List<String> = emptyList()
  2. Modify AndroidArgs so that it's a data class, allowing us to use things like copy
  3. AndroidArgs needs to be updated to allow the app parameter to be adjusted on the fly
  4. Update AndroidTestRunner to upload the new APKs in the same way the main app apk is uploaded today.
  5. Update AndroidTestRunner to iterate through the --module-apks and pass each to the AndroidArgs before creating a GcAndroidTestMatrix for the shards created in each additional apk.

The changes to AndroidTestRunner would be something along these lines:

suspend fun runTests(androidArgs: AndroidArgs): MatrixMap = coroutineScope {
    val (stopwatch, runGcsPath) = GenericTestRunner.beforeRunTests(androidArgs)

    val jobs = arrayListOf<Deferred<TestMatrix>>()

    // Add all the app tests. 
    jobs.addAll(runTestsInternal(androidArgs, runGcsPath))

    // Add any submodule tests as well.
    val resolvedModuleUris = resolveSubmoduleApks(androidArgs, runGcsPath)
    resolvedModuleUris.forEach { apk ->
        // Submodules package everything they need in the test apk, we can use any random app apk
        val moduleArgs = androidArgs.copy(testApk = apk)

        jobs.addAll(runTestsInternal(moduleArgs, runGcsPath))
    }

    GenericTestRunner.afterRunTests(jobs.awaitAll(), runGcsPath, stopwatch, androidArgs)
}

Things to consider..

  1. I'm not really sure modifying the AndroidArgs like this is the best option, but not doing it this way requires that we also modify the sharding logic so that we can track which APK a given test is part of so that a shard does not span more than one apk. Since the sharding logic is built into the AndroidArgs, copying them and allowing it to re-run against a new APK seems to work.

I'd like to work on this, so please let me know if anyone has better ideas on how this might be implemented. I've sort of hacked together the above solution - it's very hacky but it does work and I could clean it up for a PR if it sounds good.

@runningcode

This comment has been minimized.

Copy link
Contributor

runningcode commented Mar 1, 2019

Sort of related to this: runningcode/fladle#7 (comment)

One workaround is to use fladle and apply the fladle plugin to every module. You can then run a gradle command which runs all the tests in one gradle command.

@inktomi

This comment has been minimized.

Copy link
Author

inktomi commented Mar 1, 2019

Ah, yea - is that a supported use case though? It seems like you'd end up needing to deal with a lot of chunks of results - one per execution. I was trying to think of a solution that we could build inside Flank to do the same thing as fladle#7 is doing via separate invocations.

@bootstraponline

This comment has been minimized.

Copy link
Contributor

bootstraponline commented Mar 1, 2019

I'm open to thinking about how to solve this in Flank. In the best case situation though, this is a feature request for FTL. If FTL had API support for the use case then Flank integration is trivial.

@inktomi

This comment has been minimized.

Copy link
Author

inktomi commented Mar 1, 2019

I completely agree - I can raise an issue on their tracker and I'll link it here for more discussion.

@inktomi

This comment has been minimized.

Copy link
Author

inktomi commented Mar 1, 2019

I opened a discussion on https://issuetracker.google.com/issues/122327486 since there's not an exsisting issue and this one is very much related. I also mentioned it in the test-lab Firebase slack.

@inktomi

This comment has been minimized.

Copy link
Author

inktomi commented Mar 6, 2019

In talking to William Hester it seems like this is not something that FTL itself can support in the near future. I think it makes sense to build this on top of FTL, or as part of Fladle - let me know, I'm happy to open a PR.

@bootstraponline

This comment has been minimized.

Copy link
Contributor

bootstraponline commented Mar 6, 2019

Let's build this into flank. Let me know when you have some time to chat about the design

@bootstraponline

This comment has been minimized.

Copy link
Contributor

bootstraponline commented Mar 7, 2019

For the design, what do you think of this?

flank:
  additional-apks:
    - app: ../test_app/apks/app-debug2.apk
      test: ../test_app/apks/app-debug-androidTest2.apk
    - test: ../test_app/apks/app-debug-androidTest3.apk

If app is omitted then we'll use the gcloud: app: ....

This allows aggregating multiple runs which I think is a more general solution than only allowing a different test apk.

@winterDroid

This comment has been minimized.

Copy link
Contributor

winterDroid commented Mar 7, 2019

Are you planning to support also multiple apps? Or just one app and multiple test apks?

@winterDroid

This comment has been minimized.

Copy link
Contributor

winterDroid commented Mar 7, 2019

It seems gcloud beta does support additional APKs.

@bootstraponline

This comment has been minimized.

Copy link
Contributor

bootstraponline commented Mar 7, 2019

It seems gcloud beta does support additional APKs.

gcloud supports installing additional APKs but they don't run them.

Are you planning to support also multiple apps? Or just one app and multiple test apks?

I'm thinking if we support this, then it should be for an array of app/test pairs. If the app is omitted then we'll use the default app for the provided test apk.

@inktomi

This comment has been minimized.

Copy link
Author

inktomi commented Mar 7, 2019

The additional apks from FTL are useful only for allowing you to fill out an inter-app dependency - not for providing additional tests in a secondary apk. The testing APIs offered by FTL via gcloud / rest are targeted at the APK you upload as the main app/test apk. This is one of the topics I discussed with the FTL team.

I think if we want to allow multiple app definitions then we should think about making those a Pair internally so that we can track the main app/test combination. app is a required parameter to run tests in FTL, so we always need to know which one to use even if we make parallel internal executions.

Android libraries package all their logic and tests into a single apk, which means we can use any random app with a library test apk and tests will still execute. That's not the case with an Application module test apk, which requires the matching app.

@bootstraponline

This comment has been minimized.

Copy link
Contributor

bootstraponline commented Mar 7, 2019

Discussed on slack to call this additional-app-test-apks: to differentiate from FTL's --additional-apks which installs but doesn't run the apks.

@bootstraponline bootstraponline added this to In progress in Flank v5 Mar 8, 2019

@bootstraponline bootstraponline referenced a pull request that will close this issue Apr 10, 2019

Open

Add additional-app-test-apks support #542

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.