Continuous (Rendering) Benchmarking #1886
Comments
|
Simple AWS Lambda for collecting results set up (maplibre/ci-runners#12). Source here: https://github.com/maplibre/ci-runners/tree/main/mln-lambda I want to add the rendering benchmark to the Android instrumentation tests. AWS Device Farm allows setting a test filter, I think this syntax is supported. https://developer.android.com/reference/androidx/test/runner/AndroidJUnitRunner#execution-options: When the UI tests are run the benchmark should be excluded, when the benchmark is run only the benchmark test should be run. |
|
I don't want to add the needed secrets to the app at build time, because then unprivileged workflows would need access to them, or I would need to build in a priveleged workflow. There is a way to specify extra data when running the test on AWS Device Farm, but it doesn't seem to work. That is, this doesn't print anything: val appContext = InstrumentationRegistry.getInstrumentation().targetContext
val externalFiles = appContext.getExternalFilesDir(null)
externalFiles?.listFiles()?.forEach {
println("listFiles, externalFilesDir: $it")
}
appContext.filesDir.listFiles()?.forEach {
println("listFiles, filesDir: $it")
}I suspect that AWS Device Farm has not been updated to work with scoped storage yet. https://docs.aws.amazon.com/devicefarm/latest/developerguide/how-to-create-test-run.html Edit: I can actualy access |
|
Super close now... During a manual run on AWS Device Farm I was able to read from /storage/emulated/0, but now that the benchmark is running on CI I am hit with. |
|
I might need to use a custom test environment and use this: https://developer.android.com/training/data-storage/manage-all-files#enable-manage-external-storage-for-testing |
|
Tests with custom test environment fail with !!! FAILED BINDER TRANSACTION !!! Details
Device 00:01.983 1302 |
only works for debug builds. 😮💨 I don't know how I can get access to the external data needed for the benchmark. Tried this hack. Works locally but not on AWS Device Farm. https://stackoverflow.com/questions/70366670/android-os-11-pushing-data-files-to-an-applications-scoped-storage |
The iOS and Android rendering benchmarks should be run on CI.
Like all our instrumented tests we will run them on AWS Device Farm. I want to have to use one less powerful device, a modern mid-tier device and modern top-tier device. Very old devices are not available, I selected these (we can always expand / change them later). For Android:
For iOS:
HTTP API
The current instrumentation tests just poll AWS Device Farm until the test has completed. We do not pull any data from the device and I could not get this functionality to work thus far. For the benchmarks it is essential that we get the results from the device. I want to set up a simple database-backed HTTP API where the device sends the results after the benchmark is completed. This has the added benefit that we have a place to track results over time.
Workflow
The new benchmarking tests will be added to the existing workflows. The PR number, git SHA and HTTP API endpoint (with auth info) should be baked into the test app at build time.
After the normal workflows have completed a check with a button is created (called a requested action) that allows kicking off the benchmark workflow.
The benchmark workflow will retrieve the previously built tests that were stored as artifacts. It will create a new check to indicate the benchmarks are running and send them off to AWS Device Farm. After the benchmarks have completed the workflow will update the status of the check from running to passed (if the benchmark ran through) or failed (if something went wrong). The device will send the results to the HTTP API with a JSON payload that will look something like this:
{ "platform": "android", "deviceInfo": { "manufacturer": "Samsung", "model": "..." }, "benchmarkResults": { "maptilerBasic": { "avgFps": 123.0, "avgFrameEncodingTime": 123.0, "low1pFps": 60.0, "avgFps": 120.0 }, "facebook": { "avgFps": 123.0, "avgFrameEncodingTime": 123.0, "low1pFps": 60.0, "avgFps": 120.0 } }, "sha": "a858e7b4dedcf04aa6e011663e4db774631764c9", "prNumber": "222" }So: the benchmarking results (split out per style), information about the device and information about the related git commit.
When the HTTP API receives a result, it updates (or creates) the comment on the PR with the results. I can also add a comparison with main. It might be helpful to look up the results of the most recent parent commit already part of main instead, but just comparing with the latest in main is OK at first.
Cost-allowing we might want to run the benchmarks for every PR and add a failed check if a significant performance regression is determined.
Android
iOS
GitHub Actions
HTTP API
The text was updated successfully, but these errors were encountered: