-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Run UI tests nightly def #17513
Run UI tests nightly def #17513
Conversation
This PR continues the work done in PR #17017, it got a first review from Johan and the approach was discussed with him. The idea is to run at least one test on the Nightly build in addition to the tests that run in the debug build so that if it fails we know something is not correct with the Nightly build that is used by the cron job to uplift it to the playstore. There will be a new task added The UI test on Nightly will be run as the existing UI tests, per push in master. |
try_task_config.json
Outdated
@@ -0,0 +1,7 @@ | |||
{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This file will be removed, just added to see that the ui-test-x86-nightly task is created successfully
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I ran taskgraph-diff
against this to see what it changes in the taskgraph. The diff output is here.
Overall this looks good. I do see the on-push task, the nightly task, and the signing task. I think the other diffs are due to the difficulty of diffing large json blobs against each other.
Thanks!
@st3fan just to let you know about this and hear your feedback if you think this is useful or not and in case we need to/is good to add some mergify rules... Thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
taskcluster/ci/ui-test/kind.yml
Outdated
signing-android-test: signing-android-test-nightly | ||
run: | ||
commands: | ||
- [automation/taskcluster/androidTest/ui-test.sh, x86-start-nightly-test, app.apk, android-test.apk, '50'] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we want 50 here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think so 😓 It would be enough with one since we are running only one test atm. According to the comments, that could be -1:
echo "To run UI tests on ARM device shard (1 test / shard)"
echo "$ ui-test.sh arm64-v8a -1"
3e26477
to
e123fc2
Compare
Codecov Report
@@ Coverage Diff @@
## master #17513 +/- ##
============================================
+ Coverage 32.50% 32.56% +0.06%
+ Complexity 1320 1314 -6
============================================
Files 458 455 -3
Lines 19227 19089 -138
Branches 2698 2669 -29
============================================
- Hits 6249 6217 -32
+ Misses 12463 12382 -81
+ Partials 515 490 -25 Continue to review full report at Codecov.
|
f5b2e9d
to
3fbd713
Compare
3fbd713
to
79850a3
Compare
This reverts commit 06c1734.
* run-ui-test-nightly * Run ui tests on nightly build * set correct number of shard according to tests running * fix typo * removing testing file Co-authored-by: isabel rios <isabelrios@MBP-de-isabel-2.home>
…ile#17761) This reverts commit 06c1734.
This PR continues the work to have the UI tests running against the correct and each build, beta, nightly or debug...
Pull Request checklist
To download an APK when reviewing a PR: