PTS improvements #566

ribafish · 2023-08-18T13:12:09Z

Enabled Predictive Test Selection (PTS) by default.
Set the FAST PTS profile for testAndroid${version} test tasks and STANDARD profile for "regular" test task.
Set RELEVANT_TESTS PTS mode for PRs (pre-merge) and REMAINING_TESTS for main (post-merge) branch builds.
Enabled Verify Build action for all PRs, not just the ones targeting main

…sion}` tasks. Also set REMAINING_TESTS mode for `main` branch builds on CI.

runningcode

I like where this is going, but I don't like the extra complexity that adding the PTS Mode adds to this already complex CI build. Is there a way we can do it without adding the extra dependency on the generate_versions_pts job?

Second thing, if we enable PTS by default, let's be sure to disable it for release jobs.

release/changes.md

ribafish · 2023-08-21T13:58:05Z

I like where this is going, but I don't like the extra complexity that adding the PTS Mode adds to this already complex CI build. Is there a way we can do it without adding the extra dependency on the generate_versions_pts job?

I don't see a different way to be honest, at least not a simpler one. This seems pretty straightforward to me though, If this run has github.ref_name as main, then it's REMAINING_TESTS, otherwise it's RELATIVE_TESTS. The only thing to note is that when a PR is opened and triggers this action, the ref_name will hold the automatically created "merge" branch, so if it's a PR, this won't be true, even if base branch is main.

Second thing, if we enable PTS by default, let's be sure to disable it for release jobs.

The way it is set up at the moment, the only way I see that wouldn't create a completely separate workflow for release jobs (which might also make sense to do tbh) is to use workflow_dispatch.inputs with default set to false and trigger it without it (by default), while still allowing to turn it on if wanted when running it manually and set it to true if it's empty (which happens when the action is started from push to main or PR)

…red via `workflow_dispatch`)

runningcode · 2023-08-22T10:22:16Z

.github/workflows/build-verification.yml

+            -PisPTSEnabled=${{ github.event.inputs.pts_enabled }}
        env:
          GRADLE_ENTERPRISE_ACCESS_KEY: ${{ secrets.GE_SOLUTIONS_ACCESS_TOKEN }}
-          ORG_GRADLE_PROJECT_isPTSEnabled: ${{ github.ref_name != 'main' }}


Is it possible to use variable here to set the remaining tests mode inline instead of having an action to set it?

something like pts.mode: if ${{ github.ref_name == 'main' }}; then 'REMAINING_TESTS'; else 'RELEVANT_TESTS' and then pass that value?

if statements can't be used in the ENV part of actions, but there are some tertiary workarounds, one of which I've applied. It does need to be duplicated on both test jobs though...

Thanks for looking in to this. While there is some duplication in this solution, i prefer the simplicity of it. It also makes it easier to use as an example for customers.

…l workflow

* main: Use AGP 8.1.1 for main build Run tests against AGP 8.1.1 # Conflicts: # release/changes.md

runningcode · 2023-08-22T16:05:31Z

release/changes.md

@@ -1,2 +1,2 @@
 - [NEW] Run tests against AGP 8.2.0-alpha16
- [NEW] Run tests against AGP 8.1.1


why is this showing a diff? is there a whitespace change?

Yeah, I think it was without the newline at the end before. It's weird though, as I resolved it to the main branch variant of it when merging..

runningcode · 2023-08-22T16:06:46Z

.github/workflows/build-verification.yml

+            -PisPTSEnabled=${{ github.event.inputs.pts_enabled }}
        env:
          GRADLE_ENTERPRISE_ACCESS_KEY: ${{ secrets.GE_SOLUTIONS_ACCESS_TOKEN }}
-          ORG_GRADLE_PROJECT_isPTSEnabled: ${{ github.ref_name != 'main' }}


Thanks for looking in to this. While there is some duplication in this solution, i prefer the simplicity of it. It also makes it easier to use as an example for customers.

.github/workflows/build-verification.yml

runningcode

Alright as discussed and as per the manual, Let's use remaining tests mode for all pre-release cases instead of disabling PTS for those cases.

… default (also for staging builds).

runningcode

Looks great. I like the simplicity of this approach.

runningcode · 2023-08-28T08:46:53Z

.github/workflows/build-verification.yml

        run: echo "matrix=$(jq -cM '.supportedVersions | keys' src/main/resources/versions.json | sed -e 's/\./_/g' -e 's/-/_/g')" >> $GITHUB_OUTPUT
      - name: debug
-        run: echo ${{ steps.setup-matrix.outputs.matrix }}
+        run: echo "matrix=${{ steps.setup-matrix.outputs.matrix }}"


just curious, why this change?

Oh, it's still left from the first variant where I set the PTS mode in that job, but I still left it in as it's more verbose when looking at logs. I can remove this change if wanted.

let's leave it how it originally was to simplify and reduce the changeset.

Enabled PTS by default and set the FAST profile for `testAndroid${ver…

f5b2cc8

…sion}` tasks. Also set REMAINING_TESTS mode for `main` branch builds on CI.

runningcode suggested changes Aug 18, 2023

View reviewed changes

release/changes.md Outdated Show resolved Hide resolved

ribafish added 2 commits August 21, 2023 16:12

Set PTS to disabled by default when running manually (workflow trigge…

68ac58e

…red via `workflow_dispatch`)

Removed non user faacing changes from changes.md

15af1aa

ribafish requested a review from runningcode August 22, 2023 09:57

runningcode reviewed Aug 22, 2023

View reviewed changes

ribafish added 3 commits August 22, 2023 16:56

Reworked PTS mode to be set in ENV parts of the build-verification.ym…

4929580

…l workflow

Cleanup

277464d

Merge branch 'main' into gk/ptsImprovements

c2239b6

* main: Use AGP 8.1.1 for main build Run tests against AGP 8.1.1 # Conflicts: # release/changes.md

ribafish requested a review from runningcode August 22, 2023 15:03

runningcode reviewed Aug 22, 2023

View reviewed changes

runningcode approved these changes Aug 23, 2023

View reviewed changes

runningcode reviewed Aug 23, 2023

View reviewed changes

ribafish and others added 2 commits August 28, 2023 10:30

Removed the workflow_dispatch.input, which was used to disable PTS by…

21ffe45

… default (also for staging builds).

Merge branch 'main' into gk/ptsImprovements

7d2bc51

runningcode approved these changes Aug 28, 2023

View reviewed changes

runningcode reviewed Aug 28, 2023

View reviewed changes

ribafish merged commit 205da69 into main Aug 29, 2023

ribafish deleted the gk/ptsImprovements branch August 29, 2023 10:26

		@@ -1,2 +1,2 @@
		- [NEW] Run tests against AGP 8.2.0-alpha16
		- [NEW] Run tests against AGP 8.1.1

PTS improvements #566

PTS improvements #566

Uh oh!

Conversation

ribafish commented Aug 18, 2023

Uh oh!

runningcode left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ribafish commented Aug 21, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

runningcode left a comment

Choose a reason for hiding this comment

Uh oh!

runningcode left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants