changes to support test sharding of test cases across devices in a device pool #32
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 8 commits
May 21, 2020 01:52
…vice pool
Notes
PROBLEM STATEMENT :
There are situation where consumers are more interested for getting the faster test execution results rather than executing same set of tests across different devices.
SOLUTION:
This CR will be addressing the above issue by providing users additional arguments like splitTestsAcrossDevices and testSpecFiles (under DeviceState). With this change the config looks something like this
devicefarm {
// Required. The Project must already exist. You can create a project in the AWS console.
projectName "EspressoTests" // required: Must already exist.
// Optional. Defaults to "Top Devices"
devicePool "AOSP Devices"
// Optional. Default is 150 minutes
executionTimeoutMinutes 20
// Optional. Set to "off" if you want to disable device video recording during a run. Default is "on"
videoRecording "on"
// Optional. Set to "off" if you want to disable device performance monitoring during a run. Default is "on"
performanceMonitoring "on"
// Required. You must specify either accessKey and secretKey OR roleArn. roleArn takes precedence.
authentication {
roleArn "xxxx"
}
// Optional block. Radios default to 'on' state, all parameters are optional
devicestate {
extraDataZipFile null // or ‘null’ if you have no extra data. Default is null.
auxiliaryApps files(file("src/androidTest/auxiliaryApps/orchestrator-1.2.0.apk"),
file("src/androidTest/auxiliaryApps/test-services-1.2.0.apk")) // or ‘files()’ if you have no auxiliary apps. Default is an empty list.
testSpecFiles files(file("src/androidTest/yaml/EURegionNewUserTests.yml"),
file("src/androidTest/yaml/Core_Playback_People_Places_Tests.yml"),
file("src/androidTest/yaml/JPRegionNewUserTests.yml"),
file("src/androidTest/yaml/NewUserScenarios.yml"))// or ‘files()’ if you have no auxiliary apps. Default is an empty list.
wifi "on"
bluetooth "off"
gps "off"
nfc "on"
latitude 47.6204 // default
longitude -122.3491 // default
}
testSharding true
// Instrumentation
instrumentation {
// Optional. See the AWS Developer docs for filter rules
}
}
So assuming user has multiple devices in the device pool we try to create separate run for each testSpec File provided by the user. In the above example user will see 4 different runs. But user should build their own mechanism on parsing the over all test runs since they get multiple test runs as part of this change. This change is targeting to get faster test results.
PROBLEM STATEMENT 2:
We are facing nullPointerException for accessKey even when we provide roleArn.
SOLUTION:
DeviceFarmClientFactory is updated to which would work as expected if user provide just roleArn it would fetch the credentials and construct AWSDeviceFarm client for the plugin.
Testing
Verified the changes in device farm by triggering tests from local
…vice pool
Notes
PROBLEM STATEMENT :
There are situation where consumers are more interested for getting the faster test execution results rather than executing same set of tests across different devices.
SOLUTION:
This CR will be addressing the above issue by providing users additional arguments like splitTestsAcrossDevices and testSpecFiles (under DeviceState). With this change the config looks something like this
devicefarm {
// Required. The Project must already exist. You can create a project in the AWS console.
projectName "EspressoTests" // required: Must already exist.
// Optional. Defaults to "Top Devices"
devicePool "AOSP Devices"
// Optional. Default is 150 minutes
executionTimeoutMinutes 20
// Optional. Set to "off" if you want to disable device video recording during a run. Default is "on"
videoRecording "on"
// Optional. Set to "off" if you want to disable device performance monitoring during a run. Default is "on"
performanceMonitoring "on"
// Required. You must specify either accessKey and secretKey OR roleArn. roleArn takes precedence.
authentication {
roleArn "xxxx"
}
// Optional block. Radios default to 'on' state, all parameters are optional
devicestate {
extraDataZipFile null // or ‘null’ if you have no extra data. Default is null.
auxiliaryApps files(file("src/androidTest/auxiliaryApps/orchestrator-1.2.0.apk"),
file("src/androidTest/auxiliaryApps/test-services-1.2.0.apk")) // or ‘files()’ if you have no auxiliary apps. Default is an empty list.
testSpecFiles files(file("src/androidTest/yaml/EURegionNewUserTests.yml"),
file("src/androidTest/yaml/Core_Playback_People_Places_Tests.yml"),
file("src/androidTest/yaml/JPRegionNewUserTests.yml"),
file("src/androidTest/yaml/NewUserScenarios.yml"))// or ‘files()’ if you have no auxiliary apps. Default is an empty list.
wifi "on"
bluetooth "off"
gps "off"
nfc "on"
latitude 47.6204 // default
longitude -122.3491 // default
}
testSharding true
// Instrumentation
instrumentation {
// Optional. See the AWS Developer docs for filter rules
}
}
So assuming user has multiple devices in the device pool we try to create separate run for each testSpec File provided by the user. In the above example user will see 4 different runs. But user should build their own mechanism on parsing the over all test runs since they get multiple test runs as part of this change. This change is targeting to get faster test results.
PROBLEM STATEMENT 2:
We are facing nullPointerException for accessKey even when we provide roleArn.
SOLUTION:
DeviceFarmClientFactory is updated to which would work as expected if user provide just roleArn it would fetch the credentials and construct AWSDeviceFarm client for the plugin.
Testing
Verified the changes in device farm by triggering tests from local
…pool and fix for access key null
Notes
PROBLEM STATEMENT :
There are situation where consumers are more interested for getting the faster test execution results rather than executing same set of tests across different devices.
SOLUTION:
This CR will be addressing the above issue by providing users additional arguments like splitTestsAcrossDevices and testSpecFiles (under DeviceState). With this change the config looks something like this
devicefarm {
// Required. The Project must already exist. You can create a project in the AWS console.
projectName "EspressoTests" // required: Must already exist.
// Optional. Defaults to "Top Devices"
devicePool "AOSP Devices"
// Optional. Default is 150 minutes
executionTimeoutMinutes 20
// Optional. Set to "off" if you want to disable device video recording during a run. Default is "on"
videoRecording "on"
// Optional. Set to "off" if you want to disable device performance monitoring during a run. Default is "on"
performanceMonitoring "on"
// Required. You must specify either accessKey and secretKey OR roleArn. roleArn takes precedence.
authentication {
roleArn "xxxx"
}
// Optional block. Radios default to 'on' state, all parameters are optional
devicestate {
extraDataZipFile null // or ‘null’ if you have no extra data. Default is null.
auxiliaryApps files(file("src/androidTest/auxiliaryApps/orchestrator-1.2.0.apk"),
file("src/androidTest/auxiliaryApps/test-services-1.2.0.apk"),
file("src/androidTest/auxiliaryApps/CDPAutomationHelperApp.apk")) // or ‘files()’ if you have no auxiliary apps. Default is an empty list.
testSpecFiles files(file("src/androidTest/yaml/EURegionNewUserTests.yml"),
file("src/androidTest/yaml/Core_Playback_People_Places_Tests.yml"),
file("src/androidTest/yaml/JPRegionNewUserTests.yml"),
file("src/androidTest/yaml/NewUserScenarios.yml"))// or ‘files()’ if you have no auxiliary apps. Default is an empty list.
wifi "on"
bluetooth "off"
gps "off"
nfc "on"
latitude 47.6204 // default
longitude -122.3491 // default
}
testShardingEnabled true
// Instrumentation
instrumentation {
// Optional. See the AWS Developer docs for filter rules
}
}
So assuming user has multiple devices in the device pool we try to create separate run for each testSpec File provided by the user. In the above example user will see 4 different runs. But user should build their own mechanism on parsing the over all test runs since they get multiple test runs as part of this change. This change is targeting to get faster test results.
PROBLEM STATEMENT 2:
We are facing nullPointerException for accessKey even when we provide roleArn.
SOLUTION:
DeviceFarmClientFactory is updated to which would work as expected if user provide just roleArn it would fetch the credentials and construct AWSDeviceFarm client for the plugin.
Testing
Verified the changes in device farm
anandkiran2007
changed the title
|
It's funny that the very useful functionality implemented in this PR is still not merged or even commented on. @anandkiran2007 good luck man. Just try a better solution for your needs - https://marathonlabs.io |
anandkiran2007
commented
May 15, 2023
|
@anandkiran2007 Sorry for the delay on this. I am taking a look at this and will revert soon. |
|
Can this be closed if the changes can be merged ? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Notes
PROBLEM STATEMENT:
In certain situations, consumers prioritize obtaining faster test execution results over running the same set of tests across different devices.
SOLUTION:
This change aims to tackle the aforementioned issue by introducing additional arguments for users, namely splitTestsAcrossDevices and testSpecFiles (under DeviceState). With this enhancement, the configuration will be modified as follows: [Rephrase the configuration example based on the provided information.]
devicefarm {
In the scenario where the user has multiple devices in the device pool, the updated implementation aims to create separate runs for each testSpec File provided by the user. As a result, the user will observe four different runs in the given example. However, it is important to note that the responsibility of parsing the overall test runs and managing the results will still lie with the user. This change is intended to enhance the speed of obtaining test results, empowering the user to implement their own mechanism for parsing and managing the multiple test runs.
PROBLEM STATEMENT 2:
We are facing nullPointerException for accessKey even when we provide roleArn.
SOLUTION:
The updated version of DeviceFarmClientFactory now allows users to provide only the roleArn, and it will automatically fetch the necessary credentials and create an AWSDeviceFarm client for the plugin, ensuring proper functionality.
Testing
Verified the changes in device farm