Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ews-build] Split iOS layout test queue into no-WPT / only-WPT configurations #12032

Merged
merged 1 commit into from Apr 6, 2023
Merged

[ews-build] Split iOS layout test queue into no-WPT / only-WPT configurations #12032

merged 1 commit into from Apr 6, 2023

Conversation

ryanhaddad
Copy link
Contributor

@ryanhaddad ryanhaddad commented Mar 28, 2023 •
edited by webkit-early-warning-system

ac46799

[ews-build] Split iOS layout test queue into no-WPT / only-WPT configurations
https://bugs.webkit.org/show_bug.cgi?id=254559
rdar://107291888

Reviewed by Jonathan Bedard and Aakash Jain.

In order to reduce the turnaround time for iOS layout test results in EWS, split the tests
into no-WPT / only-WPT configurations that can run in parallel.

* Tools/CISupport/ews-app/ews/common/buildbot.py:
(Buildbot):
* Tools/CISupport/ews-app/ews/common/github.py:
(GitHubEWS):
* Tools/CISupport/ews-app/ews/views/statusbubble.py:
* Tools/CISupport/ews-build/config.json:
* Tools/CISupport/ews-build/factories.py:
(iOSTestsOnlyWPTFactory):
(iOSTestsNoWPTFactory):
* Tools/CISupport/ews-build/steps.py:
(RunWebKitTestsInStressMode):
(RunWebKitTestsOnlyWPT):
(RunWebKitTestsOnlyWPT.__init__):
(RunWebKitTestsOnlyWPT.setLayoutTestCommand):
(RunWebKitTestsNoWPT):
(RunWebKitTestsNoWPT.__init__):
(RunWebKitTestsNoWPT.setLayoutTestCommand):
* Tools/CISupport/ews-build/steps_unittest.py:
(TestRunWebKitTestsNoWPT):
(TestRunWebKitTestsNoWPT.setUp):
(TestRunWebKitTestsNoWPT.tearDown):
(TestRunWebKitTestsNoWPT.configureStep):
(TestRunWebKitTestsNoWPT.test_success):

Canonical link: https://commits.webkit.org/262673@main

5d8a4fe

Misc iOS, tvOS & watchOS macOS Linux Windows
✅ 🧪 style ✅ 🛠 ios ✅ 🛠 mac ✅ 🛠 wpe   🛠 wincairo
✅ 🧪 bindings ✅ 🛠 ios-sim ✅ 🛠 mac-AS-debug   🧪 wpe-wk2
✅ 🧪 webkitperl   🧪 ios-wk2   🧪 api-mac ✅ 🛠 gtk
  🧪 api-ios   🧪 mac-wk1   🧪 gtk-wk2
  🛠 tv   🧪 mac-wk2   🧪 api-gtk
  🛠 tv-sim   🧪 mac-AS-debug-wk2
✅ 🧪 services   🛠 watch   🧪 mac-wk2-stress
✅ 🛠 watch-sim
✅ 🛠 🧪 unsafe-merge

@ryanhaddad ryanhaddad self-assigned this Mar 28, 2023
@ryanhaddad ryanhaddad added the Tools / Tests Tools in the Tools directory, build issues, test infrastructure, and bugs in test cases label Mar 28, 2023
@webkit-early-warning-system
Copy link
Collaborator

EWS run on previous version of this PR (hash e5f1415)

Misc iOS, tvOS & watchOS macOS Linux Windows
✅ 🧪 style ✅ 🛠 ios   🛠 mac   🛠 wpe   🛠 wincairo
  🧪 bindings   🛠 ios-sim   🛠 mac-AS-debug   🧪 wpe-wk2
✅ 🧪 webkitperl   🧪 ios-wk2   🧪 api-mac   🛠 gtk
  🧪 api-ios   🧪 mac-wk1   🧪 gtk-wk2
  🛠 tv   🧪 mac-wk2   🧪 api-gtk
  🛠 tv-sim   🧪 mac-AS-debug-wk2
❌ 🧪 services   🛠 watch   🧪 mac-wk2-stress
  🛠 watch-sim

@ryanhaddad ryanhaddad added the skip-ews Applied to prevent a change from being run on EWS label Mar 28, 2023
nt1m
nt1m reviewed Mar 28, 2023
View reviewed changes
Tools/CISupport/ews-build/steps_unittest.py Outdated Show resolved Hide resolved
Tools/CISupport/ews-build/steps_unittest.py Outdated Show resolved Hide resolved
Tools/CISupport/ews-build/steps_unittest.py Outdated Show resolved Hide resolved
Tools/CISupport/ews-build/steps_unittest.py Outdated Show resolved Hide resolved
@ryanhaddad ryanhaddad removed the skip-ews Applied to prevent a change from being run on EWS label Mar 28, 2023
@ryanhaddad ryanhaddad force-pushed the eng/ews-build-Split-iOS-layout-test-queue-into-no-WPT--only-WPT-configurations branch from e5f1415 to 07f4360 Compare March 28, 2023 15:34
@webkit-early-warning-system
Copy link
Collaborator

EWS run on previous version of this PR (hash 07f4360)

Misc iOS, tvOS & watchOS macOS Linux Windows
✅ 🧪 style ✅ 🛠 ios ✅ 🛠 mac ✅ 🛠 wpe ✅ 🛠 wincairo
✅ 🧪 bindings ✅ 🛠 ios-sim ✅ 🛠 mac-AS-debug   🧪 wpe-wk2
✅ 🧪 webkitperl   🧪 ios-wk2 ✅ 🧪 api-mac ✅ 🛠 gtk
  🧪 api-ios   🧪 mac-wk1   🧪 gtk-wk2
✅ 🛠 tv ✅ 🧪 mac-wk2   🧪 api-gtk
✅ 🛠 tv-sim   🧪 mac-AS-debug-wk2
❌ 🧪 services ✅ 🛠 watch ✅ 🧪 mac-wk2-stress
✅ 🛠 watch-sim

@ryanhaddad ryanhaddad force-pushed the eng/ews-build-Split-iOS-layout-test-queue-into-no-WPT--only-WPT-configurations branch from 07f4360 to d9ed130 Compare March 28, 2023 16:28
@webkit-early-warning-system
Copy link
Collaborator

EWS run on previous version of this PR (hash d9ed130)

Misc iOS, tvOS & watchOS macOS Linux Windows
✅ 🧪 style ✅ 🛠 ios ✅ 🛠 mac ✅ 🛠 wpe   🛠 wincairo
✅ 🧪 bindings ✅ 🛠 ios-sim ✅ 🛠 mac-AS-debug   🧪 wpe-wk2
✅ 🧪 webkitperl   🧪 ios-wk2   🧪 api-mac   🛠 gtk
  🧪 api-ios   🧪 mac-wk1   🧪 gtk-wk2
✅ 🛠 tv   🧪 mac-wk2   🧪 api-gtk
  🛠 tv-sim   🧪 mac-AS-debug-wk2
❌ 🧪 services ✅ 🛠 watch   🧪 mac-wk2-stress
✅ 🛠 watch-sim

@ryanhaddad ryanhaddad added skip-ews Applied to prevent a change from being run on EWS and removed skip-ews Applied to prevent a change from being run on EWS labels Mar 28, 2023
@ryanhaddad ryanhaddad force-pushed the eng/ews-build-Split-iOS-layout-test-queue-into-no-WPT--only-WPT-configurations branch from d9ed130 to a0060ea Compare March 28, 2023 16:50
@webkit-early-warning-system
Copy link
Collaborator

EWS run on previous version of this PR (hash a0060ea)

Misc iOS, tvOS & watchOS macOS Linux Windows
✅ 🧪 style ✅ 🛠 ios ✅ 🛠 mac ✅ 🛠 wpe   🛠 wincairo
✅ 🧪 bindings ✅ 🛠 ios-sim ✅ 🛠 mac-AS-debug   🧪 wpe-wk2
✅ 🧪 webkitperl   🧪 ios-wk2   🧪 api-mac   🛠 gtk
  🧪 api-ios   🧪 mac-wk1   🧪 gtk-wk2
✅ 🛠 tv   🧪 mac-wk2   🧪 api-gtk
  🛠 tv-sim   🧪 mac-AS-debug-wk2
✅ 🧪 services ✅ 🛠 watch   🧪 mac-wk2-stress
✅ 🛠 watch-sim

@ryanhaddad
Copy link
Contributor Author

ryanhaddad commented Mar 28, 2023 •
edited

Next step is to test this on the UAT instance before marking ready for review.

Update:
No WPT: https://ews-build.webkit-uat.org/#/builders/29/builds/6
Only WPT: https://ews-build.webkit-uat.org/#/builders/72/builds/6

The split worked as expected, but the failures didn't trigger retry logic, so that needs to be addressed.

@ryanhaddad ryanhaddad added skip-ews Applied to prevent a change from being run on EWS and removed skip-ews Applied to prevent a change from being run on EWS labels Mar 28, 2023
@ryanhaddad ryanhaddad force-pushed the eng/ews-build-Split-iOS-layout-test-queue-into-no-WPT--only-WPT-configurations branch from a0060ea to 3253ad2 Compare March 29, 2023 18:06
@webkit-early-warning-system
Copy link
Collaborator

EWS run on previous version of this PR (hash 3253ad2)

Misc iOS, tvOS & watchOS macOS Linux Windows
✅ 🧪 style ✅ 🛠 ios ✅ 🛠 mac   🛠 wpe ⏳ 🛠 wincairo
  🧪 bindings ✅ 🛠 ios-sim   🛠 mac-AS-debug   🧪 wpe-wk2
✅ 🧪 webkitperl   🧪 ios-wk2   🧪 api-mac   🛠 gtk
  🧪 api-ios   🧪 mac-wk1   🧪 gtk-wk2
⏳ 🛠 🧪 jsc   🛠 tv   🧪 mac-wk2   🧪 api-gtk
✅ 🛠 tv-sim   🧪 mac-AS-debug-wk2
✅ 🧪 services   🛠 watch   🧪 mac-wk2-stress
✅ 🛠 watch-sim

@ryanhaddad ryanhaddad added the skip-ews Applied to prevent a change from being run on EWS label Mar 29, 2023
@ryanhaddad
Copy link
Contributor Author

The split worked as expected, but the failures didn't trigger retry logic, so that needs to be addressed.

This was because I originally copied RunWebKitTestsInStressMode() to make these new steps, which had its own version of evaluateCommand that didn't include retry steps.

@ryanhaddad ryanhaddad removed the skip-ews Applied to prevent a change from being run on EWS label Mar 29, 2023
@ryanhaddad ryanhaddad force-pushed the eng/ews-build-Split-iOS-layout-test-queue-into-no-WPT--only-WPT-configurations branch from 3253ad2 to 179439e Compare March 29, 2023 19:12
@webkit-early-warning-system
Copy link
Collaborator

EWS run on previous version of this PR (hash 179439e)

Misc iOS, tvOS & watchOS macOS Linux Windows
✅ 🧪 style ✅ 🛠 ios   🛠 mac   🛠 wpe ⏳ 🛠 wincairo
✅ 🧪 bindings ✅ 🛠 ios-sim ✅ 🛠 mac-AS-debug   🧪 wpe-wk2
✅ 🧪 webkitperl   🧪 ios-wk2   🧪 api-mac   🛠 gtk
⏳ 🧪 api-ios   🧪 mac-wk1   🧪 gtk-wk2
⏳ 🛠 🧪 jsc   🛠 tv   🧪 mac-wk2   🧪 api-gtk
✅ 🛠 tv-sim   🧪 mac-AS-debug-wk2
✅ 🧪 services   🛠 watch   🧪 mac-wk2-stress
✅ 🛠 watch-sim

@ryanhaddad ryanhaddad marked this pull request as ready for review March 29, 2023 19:12
@ryanhaddad ryanhaddad added the skip-ews Applied to prevent a change from being run on EWS label Mar 29, 2023
@JonWBedard
Copy link
Member

The only bit I'm unsure on is the bubble/GitHub comment matrix, I can dig in and try and understand that bit if @aj062 doesn't have any thoughts on it.

@aj062
Copy link
Member

aj062 commented Mar 29, 2023

The only bit I'm unsure on is the bubble/GitHub comment matrix, I can dig in and try and understand that bit if @aj062 doesn't have any thoughts on it.

GitHub status-bubble matrix looks fine to me. Although Bugzilla status-bubbles also needs to be updated in https://github.com/WebKit/WebKit/blob/main/Tools/CISupport/ews-app/ews/views/statusbubble.py#L46

@ryanhaddad ryanhaddad removed the skip-ews Applied to prevent a change from being run on EWS label Mar 29, 2023
@ryanhaddad ryanhaddad force-pushed the eng/ews-build-Split-iOS-layout-test-queue-into-no-WPT--only-WPT-configurations branch from 179439e to 20f05ca Compare March 29, 2023 20:28
@webkit-early-warning-system
Copy link
Collaborator

EWS run on previous version of this PR (hash 20f05ca)

Misc iOS, tvOS & watchOS macOS Linux Windows
✅ 🧪 style ✅ 🛠 ios ✅ 🛠 mac ✅ 🛠 wpe ✅ 🛠 wincairo
✅ 🧪 bindings ✅ 🛠 ios-sim ✅ 🛠 mac-AS-debug   🧪 wpe-wk2
✅ 🧪 webkitperl   🧪 ios-wk2 ✅ 🧪 api-mac ✅ 🛠 gtk
  🧪 api-ios ✅ 🧪 mac-wk1   🧪 gtk-wk2
✅ 🛠 tv ✅ 🧪 mac-wk2   🧪 api-gtk
✅ 🛠 tv-sim   🧪 mac-AS-debug-wk2
✅ 🧪 services ✅ 🛠 watch ✅ 🧪 mac-wk2-stress
✅ 🛠 watch-sim

@ryanhaddad ryanhaddad added the skip-ews Applied to prevent a change from being run on EWS label Mar 29, 2023
robert-jenner
robert-jenner reviewed Apr 5, 2023
View reviewed changes
Tools/CISupport/ews-build/config.json Outdated Show resolved Hide resolved
Tools/CISupport/ews-build/config.json Outdated Show resolved Hide resolved
@ryanhaddad ryanhaddad removed the skip-ews Applied to prevent a change from being run on EWS label Apr 5, 2023
@ryanhaddad ryanhaddad force-pushed the eng/ews-build-Split-iOS-layout-test-queue-into-no-WPT--only-WPT-configurations branch from 20f05ca to 5d8a4fe Compare April 5, 2023 21:47
@webkit-early-warning-system
Copy link
Collaborator

EWS run on current version of this PR (hash 5d8a4fe)

Misc iOS, tvOS & watchOS macOS Linux Windows
✅ 🧪 style ✅ 🛠 ios ✅ 🛠 mac ✅ 🛠 wpe   🛠 wincairo
✅ 🧪 bindings ✅ 🛠 ios-sim ✅ 🛠 mac-AS-debug   🧪 wpe-wk2
✅ 🧪 webkitperl   🧪 ios-wk2   🧪 api-mac ✅ 🛠 gtk
  🧪 api-ios   🧪 mac-wk1   🧪 gtk-wk2
  🛠 tv   🧪 mac-wk2   🧪 api-gtk
  🛠 tv-sim   🧪 mac-AS-debug-wk2
✅ 🧪 services   🛠 watch   🧪 mac-wk2-stress
✅ 🛠 watch-sim
✅ 🛠 🧪 unsafe-merge

@ryanhaddad ryanhaddad added the skip-ews Applied to prevent a change from being run on EWS label Apr 5, 2023
@ryanhaddad
Copy link
Contributor Author

@aj062 @JonWBedard may I get a formal review here? I'd like to be able to land this as soon as we've got the hardware ready.

@ryanhaddad ryanhaddad added the unsafe-merge-queue Applied to send a pull request to merge-queue, but skip building and testing label Apr 6, 2023
@ryanhaddad
[ews-build] Split iOS layout test queue into no-WPT / only-WPT config…
ac46799 
…urations

https://bugs.webkit.org/show_bug.cgi?id=254559
rdar://107291888

Reviewed by Jonathan Bedard and Aakash Jain.

In order to reduce the turnaround time for iOS layout test results in EWS, split the tests
into no-WPT / only-WPT configurations that can run in parallel.

* Tools/CISupport/ews-app/ews/common/buildbot.py:
(Buildbot):
* Tools/CISupport/ews-app/ews/common/github.py:
(GitHubEWS):
* Tools/CISupport/ews-app/ews/views/statusbubble.py:
* Tools/CISupport/ews-build/config.json:
* Tools/CISupport/ews-build/factories.py:
(iOSTestsOnlyWPTFactory):
(iOSTestsNoWPTFactory):
* Tools/CISupport/ews-build/steps.py:
(RunWebKitTestsInStressMode):
(RunWebKitTestsOnlyWPT):
(RunWebKitTestsOnlyWPT.__init__):
(RunWebKitTestsOnlyWPT.setLayoutTestCommand):
(RunWebKitTestsNoWPT):
(RunWebKitTestsNoWPT.__init__):
(RunWebKitTestsNoWPT.setLayoutTestCommand):
* Tools/CISupport/ews-build/steps_unittest.py:
(TestRunWebKitTestsNoWPT):
(TestRunWebKitTestsNoWPT.setUp):
(TestRunWebKitTestsNoWPT.tearDown):
(TestRunWebKitTestsNoWPT.configureStep):
(TestRunWebKitTestsNoWPT.test_success):

Canonical link: https://commits.webkit.org/262673@main
@webkit-commit-queue webkit-commit-queue force-pushed the eng/ews-build-Split-iOS-layout-test-queue-into-no-WPT--only-WPT-configurations branch from 5d8a4fe to ac46799 Compare April 6, 2023 18:44
@webkit-commit-queue
Copy link
Collaborator

Committed 262673@main (ac46799): https://commits.webkit.org/262673@main

Reviewed commits have been landed. Closing PR #12032 and removing active labels.

@webkit-commit-queue webkit-commit-queue removed the unsafe-merge-queue Applied to send a pull request to merge-queue, but skip building and testing label Apr 6, 2023
@webkit-commit-queue webkit-commit-queue merged commit ac46799 into WebKit:main Apr 6, 2023
@ryanhaddad ryanhaddad deleted the eng/ews-build-Split-iOS-layout-test-queue-into-no-WPT--only-WPT-configurations branch April 6, 2023 21:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nt1m nt1m nt1m left review comments

@robert-jenner robert-jenner robert-jenner left review comments

@aj062 aj062 aj062 approved these changes

@JonWBedard JonWBedard JonWBedard approved these changes

Assignees

@ryanhaddad ryanhaddad

Labels
skip-ews Applied to prevent a change from being run on EWS Tools / Tests Tools in the Tools directory, build issues, test infrastructure, and bugs in test cases
Projects
None yet
Milestone
No milestone
7 participants
@ryanhaddad @webkit-early-warning-system @JonWBedard @aj062 @webkit-commit-queue @nt1m @robert-jenner