On-device test parser #539

Malinskiy · 2021-07-06T10:27:18Z

Enabled on adam by default
Currently loses any annotations associated with tests unless a listener is attached to the test parsing, so annotation filter will not work. This is because am instrument doesn't output any annotations.

Closes #393
Depends on #538

codecov-commenter · 2021-07-07T02:29:58Z

Codecov Report

Merging #539 (01c33b3) into develop (17f57be) will decrease coverage by 1.95%.
The diff coverage is 60.44%.

@@              Coverage Diff              @@
##             develop     #539      +/-   ##
=============================================
- Coverage      66.72%   64.76%   -1.96%     
- Complexity       539      687     +148     
=============================================
  Files            121      166      +45     
  Lines           2398     3312     +914     
  Branches         325      565     +240     
=============================================
+ Hits            1600     2145     +545     
- Misses           623      880     +257     
- Partials         175      287     +112

Impacted Files	Coverage Δ
...alinskiy/marathon/config/AnalyticsConfiguration.kt	`75.00% <ø> (ø)`
...y/marathon/config/LogicalConfigurationValidator.kt	`0.00% <0.00%> (ø)`
...malinskiy/marathon/config/ScreenRecordingPolicy.kt	`100.00% <ø> (ø)`
...hon/config/environment/EnvironmentConfiguration.kt	`100.00% <ø> (ø)`
...thon/config/environment/SystemEnvironmentReader.kt	`83.33% <ø> (ø)`
...rathon/config/exceptions/ConfigurationException.kt	`33.33% <0.00%> (ø)`
...nfig/serialization/time/InstantTimeProviderImpl.kt	`100.00% <ø> (ø)`
...lization/yaml/GraphiteConfigurationDeserializer.kt	`18.18% <ø> (ø)`
...n/yaml/RetentionPolicyConfigurationDeserializer.kt	`42.10% <ø> (ø)`
...athon/config/vendor/android/AllureConfiguration.kt	`100.00% <ø> (ø)`
... and 146 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 17f57be...01c33b3. Read the comment docs.

…e and test class

… feature/device-test-parser

… feature/device-test-parser # Conflicts: # buildSrc/src/main/kotlin/Versions.kt # vendor/vendor-android/base/src/main/kotlin/com/malinskiy/marathon/android/RemoteFileManager.kt

Malinskiy · 2021-10-07T08:52:18Z

There is an experimental way to extract annotations using on-device parser: https://github.com/Malinskiy/adam/blob/feature/test-meta-listener/android-junit4-listener/src/main/kotlin/com/malinskiy/adam/junit4/android/listener/TestAnnotationProducer.kt

The idea is to attach a RunListener to the execution using am instrument -e listener TestAnnotationProducer and parse the metrics outputted by test. One of those metrics will contain a list of annotations.

In order to use this users will have to declare a androidTestImplementation dependency with such listener and add an an instrumentationArgs parameter to the test parser:

vendorConfiguration:
  type: "Android"
....
  vendor: ADAM
  testParserConfiguration:
    type: "remote"
    instrumentationArgs:
      listener: "com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer"

Then in the parser annotations will be picked up if present. To allow contract changes (e.g. annotation values) we have to version the instrumentation parameter name.

…est metrics

tagantroy · 2021-10-09T11:21:18Z

...endor-android/adam/src/main/kotlin/com/malinskiy/marathon/android/adam/OnDeviceTestParser.kt

+            for (update in channel) {
+                if (update is DeviceProvider.DeviceEvent.DeviceConnected) {
+                    val device = update.device as AdamAndroidDevice
+                    return parseTests(device, configuration, vendorConfiguration, testBundles)


First connected device will be used for test parsing? Should we have some retry logic here to prevent flakiness if connection is not stable and so on?

Sure, will do

# Conflicts: # buildSrc/src/main/kotlin/Versions.kt # cli/src/main/kotlin/com/malinskiy/marathon/cli/args/FileAndroidConfiguration.kt # cli/src/main/kotlin/com/malinskiy/marathon/cli/config/DeserializeModule.kt # configuration/src/main/kotlin/com/malinskiy/marathon/config/serialization/yaml/TestParserConfigurationDeserializer.kt # core/src/main/kotlin/com/malinskiy/marathon/execution/TestParser.kt # vendor/vendor-android/adam/src/main/kotlin/com/malinskiy/marathon/android/adam/AndroidDeviceTestRunner.kt # vendor/vendor-android/adam/src/main/kotlin/com/malinskiy/marathon/android/adam/di/Modules.kt # vendor/vendor-android/base/src/main/kotlin/com/malinskiy/marathon/android/AndroidTestParser.kt # vendor/vendor-android/base/src/main/kotlin/com/malinskiy/marathon/android/configuration/AndroidConfiguration.kt # vendor/vendor-android/base/src/test/kotlin/com/malinskiy/marathon/android/AndroidTestParserTest.kt # vendor/vendor-ios/src/main/kotlin/com/malinskiy/marathon/ios/IOSTestParser.kt # vendor/vendor-ios/src/test/kotlin/com/malinskiy/marathon/ios/IOSTestParserTest.kt # vendor/vendor-test/src/main/kotlin/com/malinskiy/marathon/test/factory/ConfigurationFactory.kt

- print message when test is ignored - trim annotations when parsing - fix test request for test without methods - fix TestRunResultsListener to report null methods if they were requested - fix cli package resource copying bug during build

…, add docs and an execution warning when no annotations are reported

… feature/device-test-parser

…athonLabs#539, MarathonLabs#551

Malinskiy added 2 commits July 6, 2021 18:05

feat(android): add named parametized test

02470b5

feat(android): test parser via device log

bb4a3bf

Malinskiy force-pushed the feature/device-test-parser branch from e09ce9b to bb4a3bf Compare July 6, 2021 10:33

fix(test): fix broken tests

df0b521

Malinskiy added 4 commits July 8, 2021 01:12

Merge branch 'develop' into feature/device-test-parser

e2ecea6

feat(android): support remote test parser

3724d38

fix(android): support for root test package and spaces in test packag…

0bf37d4

…e and test class

Merge remote-tracking branch 'origin/feature/device-test-parser' into…

7f85c88

… feature/device-test-parser

Malinskiy mentioned this pull request Oct 6, 2021

Execute cucumber features based on tags in marathon #560

Closed

Malinskiy added 3 commits October 6, 2021 18:43

Merge branch 'develop' into feature/device-test-parser

a34f975

feat(android): add android-cucumber-app sample

cd61863

Merge remote-tracking branch 'origin/feature/device-test-parser' into…

8c27255

… feature/device-test-parser # Conflicts: # buildSrc/src/main/kotlin/Versions.kt # vendor/vendor-android/base/src/main/kotlin/com/malinskiy/marathon/android/RemoteFileManager.kt

feat(android): OnDeviceTestParser support for annotations passed as t…

a9922ca

…est metrics

Malinskiy marked this pull request as ready for review October 8, 2021 04:41

Malinskiy requested review from tagantroy and Vacxe as code owners October 8, 2021 04:41

Malinskiy added 2 commits October 8, 2021 17:18

feat(android): handle annotation parameters using remote test parser

3ac79d8

Merge branch 'develop' into feature/device-test-parser

5b0eb19

tagantroy reviewed Oct 9, 2021

View reviewed changes

Malinskiy added 8 commits October 10, 2021 00:19

Merge branch 'develop' into feature/device-test-parser

6e9a402

feat(android): add gradle configuration for test parser configuration…

1d82829

…, add docs and an execution warning when no annotations are reported

Merge remote-tracking branch 'origin/feature/device-test-parser' into…

25f4834

… feature/device-test-parser

feat(android): remote test parser should retry in case of failure

e22db41

feat(android): change wording of the warning

f1262e5

docs(android): change wording of the note

3df1894

tagantroy previously approved these changes Oct 25, 2021

View reviewed changes

fix(core): fix tests

533ca7a

Malinskiy dismissed tagantroy’s stale review via 533ca7a October 28, 2021 02:56

Malinskiy added 2 commits October 28, 2021 14:15

fix(android): fix tests

d74d309

fix(configuration): add graphite analytics configuration

01c33b3

tagantroy approved these changes Oct 28, 2021

View reviewed changes

Malinskiy merged commit dc39786 into develop Oct 28, 2021

Malinskiy deleted the feature/device-test-parser branch October 28, 2021 23:36

Malinskiy mentioned this pull request Oct 31, 2021

Better AndroidTestParser implementation #480

Closed

idyatlov pushed a commit to badoo/marathon that referenced this pull request Feb 13, 2022

Add parameterized tests support from origin PRs MarathonLabs#538, Mar…

0b168a0

…athonLabs#539, MarathonLabs#551

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

On-device test parser #539

On-device test parser #539

Malinskiy commented Jul 6, 2021 •

edited

Loading

codecov-commenter commented Jul 7, 2021 •

edited

Loading

Malinskiy commented Oct 7, 2021

tagantroy Oct 9, 2021

Malinskiy Oct 9, 2021

On-device test parser #539

On-device test parser #539

Conversation

Malinskiy commented Jul 6, 2021 • edited Loading

codecov-commenter commented Jul 7, 2021 • edited Loading

Codecov Report

Malinskiy commented Oct 7, 2021

tagantroy Oct 9, 2021

Choose a reason for hiding this comment

Malinskiy Oct 9, 2021

Choose a reason for hiding this comment

Malinskiy commented Jul 6, 2021 •

edited

Loading

codecov-commenter commented Jul 7, 2021 •

edited

Loading