Tests/android UI tests absolute #2077
Conversation
added end2end ui tests module with base test class (early version)
added mock relay server with envelope and items assertions
app cache is deleted on test start MockRelay defaults to success response
added comments and javadoc to Benchmark module added a README
*sentry-android-ui-tests -> sentry-uitest *benchmark -> sentry-uitest-android-benchmark *end2end -> sentry-uitest-android Updated root build gradle file to ignore uitest modules from apiValidation, DistributionPlugin and aggregateJavadocs Added comments and javadoc on sentry-uitest-android module
…oad to saucelabs removed dependency between test modules
removed proguard fix typo in Hub transactionListener -> transactionProfiler
added a before and after method to optionally invoke in each iteration without being measured added benchmarking sdk functions
printing the number of cpu cores in benchmark results pass a Choreographer object instead of the Runner to the BenchmarkOperation
…olute # Conflicts: # sentry-uitest/sentry-uitest-android-benchmark/src/androidTest/java/io/sentry/uitest/android/benchmark/SentryBenchmarkTest.kt # sentry-uitest/sentry-uitest-android-benchmark/src/androidTest/java/io/sentry/uitest/android/benchmark/util/BenchmarkOperation.kt # sentry-uitest/sentry-uitest-android-benchmark/src/androidTest/java/io/sentry/uitest/android/benchmark/util/BenchmarkOperationResult.kt
…ration in SdkBenchmarkTest
…olute # Conflicts: # sentry-uitest/sentry-uitest-android-benchmark/src/androidTest/java/io/sentry/uitest/android/benchmark/SentryBenchmarkTest.kt
added ViewBinding few cleanups moved "androidx.test.runner.AndroidJUnitRunner" in Config.TestLibs.androidJUnitRunner
…ts/android-ui-tests
assertEnvelopeItem is now a <reified T>
added a couple jobs to the makefile for integration test apks
added a couple jobs to the makefile for integration test apks
added a couple jobs to the makefile for integration test apks
|
@stefanosiano let us know once this PR is ready to be reviewed since there are still commits going on, thanks. |
This pr is ready. I'm just making a few changes for the new github actions, but only on the .yml files |
...ndroid-benchmark/src/androidTest/java/io/sentry/uitest/android/benchmark/SdkBenchmarkTest.kt
Outdated
Show resolved
Hide resolved
...oid-benchmark/src/androidTest/java/io/sentry/uitest/android/benchmark/SentryBenchmarkTest.kt
Show resolved
Hide resolved
...benchmark/src/androidTest/java/io/sentry/uitest/android/benchmark/util/BenchmarkOperation.kt
Outdated
Show resolved
Hide resolved
|
@stefanosiano did a review and left a few comments, please address them and make the tests pass, so CI is green, after that we can approve & merge it, thanks for doing this. |
removed activityStartedIdlingResource from BenchmarkActivity as Espresso already waits for full activity creation when launching it changed scrollingIdlingResource from BooleanIdlingResource to CountingIdlingResource to avoid dependency on another module changed uitest module to "com.android.application", to allow apk creation to upload it to SauceLabs
|
fixed the tests and now we have them running on SauceLabs! |
| testOptions: | ||
| useTestOrchestrator: true | ||
|
|
||
| # Commenting for the moment, due to the error "Cannot install test-services-1.4.1.apk on device" on low Android versions |
There was a problem hiding this comment.
Why do we get this error? Should we fix or remove the comments?
There was a problem hiding this comment.
The errors are related to an issue with Android test orchestrator on old versions.
I will fix the issue, but in another pr, so we can understand how the sauce labs runs work with the other commits (adding devices would make the tests take longer)
📜 Description
We have benchmarks that calculate relative difference between two operations with percentages.
We are now adding absolute difference difference between two operations.
💡 Motivation and Context
We want to make sure that certain operations don't take too long. E.g. the sdk init shouldn't take more than a few milliseconds.
#skip-changelog
💚 How did you test it?
🤣
📝 Checklist
🔮 Next steps
Check other devices we can run the benchmarks on (there is still some issues with android runner on old versions). Will add them in another pr.