Skip to content

Reduce run-webkit-tests startup overhead for faster iteration#64594

Open
aproskuryakov wants to merge 1 commit intoWebKit:mainfrom
aproskuryakov:eng/Reduce-run-webkit-tests-startup-overhead-for-faster-iteration
Open

Reduce run-webkit-tests startup overhead for faster iteration#64594
aproskuryakov wants to merge 1 commit intoWebKit:mainfrom
aproskuryakov:eng/Reduce-run-webkit-tests-startup-overhead-for-faster-iteration

Conversation

@aproskuryakov
Copy link
Copy Markdown
Contributor

@aproskuryakov aproskuryakov commented May 9, 2026

90cd151

Reduce run-webkit-tests startup overhead for faster iteration
https://bugs.webkit.org/show_bug.cgi?id=314459
rdar://176618242

Reviewed by NOBODY (OOPS!).

`run-webkit-tests fast/dom/ChildNode-replaceWith.html` wall clock drops from ~5s to ~2.8s.
WebKitTestRunner itself still takes ~1s; the rest is setup overhead that this change trims.

Each of these improvements was independently measurable:
- LayoutTestHelper's ~440ms color-profile install now overlaps with test discovery instead
of blocking _set_up_run.
- configuration_for_upload (which pulls in requests, ~80ms) runs only when --report is passed.
- webkit-build-directory perl calls: 4 → 1 via a class-level cache and a lazy --target
default (~500ms).
- WebKitTestRunner warmup is skipped when child_processes == 1.
- Several production import paths no longer pull in unittest.mock, webkitcorepy.mocks
submodules, the whole requests stack, or the embedded-port chain on a Mac run.
- Misc: cached inner finder, precompiled regex, lazy port-factory table.

* Tools/Scripts/webkitpy/layout_tests/models/test_expectations.py:
(TestExpectationParser._tokenize_line):
Collapse whitespace with `" ".join(s.split())` instead of a regex, and precompile _BUG_TOKEN_RE.

* Tools/Scripts/webkitpy/port/base.py:
(Port.start_helper_async):
(Port.wait_for_helper_ready):
New no-op defaults; MacPort overrides. Splitting start_helper into a kick-off + join lets
the caller overlap the helper with other work.
(Port.stop_helper):
Reset _helper_start_attempted so a later run isn't mistaken for a failed one.

* Tools/Scripts/webkitpy/port/config.py:
(Config._build_directories):
(Config._clear_cache_for_testing):
(Config.build_directory):
Cache is class-level now (invariant within a process and sharable across Config instances)
and keyed on (port_implementation, configuration, for_host).

* Tools/Scripts/webkitpy/port/base_unittest.py:
* Tools/Scripts/webkitpy/port/config_unittest.py:
(PortTest.setUp):
(ConfigTest.setUp):
Clear the class cache between tests and between phases within test_build_directory.

* Tools/Scripts/webkitpy/port/darwin.py:
(DarwinPort.CURRENT_VERSION):
Import from version_name_map instead of literal Version(26), to avoid duplication.

* Tools/Scripts/webkitpy/port/factory.py:
(_lazy_port):
(PortFactory.PORT_CLASSES):
(PortFactory.get):
Replace the eager loop over all 14 port modules with a table mapping port-name prefix → CallByNeed.
Only the matched port is imported.
(configuration_options):
--target now defaults to None. Port.__init__ already calls set_option_default('configuration',
self.default_configuration()) after parsing, so the old eager Config.default_configuration()
call at option-definition time (one perl invocation every run) is redundant.

* Tools/Scripts/webkitpy/port/factory_unittest.py:
(FactoryTest.test_port_classes_table_consistency):
Assert each prefix in PORT_CLASSES matches the imported class's port_name so the duplication
can't silently drift.

* Tools/Scripts/webkitpy/port/mac.py:
(MacPort.setup_test_run):
Skip the warmup invocation when child_processes == 1.
(MacPort.start_helper):
(MacPort.start_helper_async):
(MacPort.wait_for_helper_ready):
Split into non-blocking popen + blocking readline. Track _ready on the Popen handle and
_helper_start_attempted on the class so wait returns False for a helper that failed to spawn.

90cd151

Misc iOS, visionOS, tvOS & watchOS macOS Linux Windows Apple Internal
✅ 🧪 style ✅ 🛠 ios ✅ 🛠 mac ✅ 🛠 wpe ✅ 🛠 win ⏳ 🛠 ios-apple
✅ 🧪 bindings ✅ 🛠 ios-sim ✅ 🛠 mac-AS-debug ✅ 🧪 wpe-wk2 ❌ 🧪 win-tests ⏳ 🛠 mac-apple
✅ 🧪 webkitperl ✅ 🧪 ios-wk2 ✅ 🧪 api-mac ✅ 🧪 api-wpe ⏳ 🛠 vision-apple
✅ 🧪 webkitpy ✅ 🧪 ios-wk2-wpt ✅ 🧪 api-mac-debug ✅ 🛠 gtk3-libwebrtc
✅ 🧪 api-ios ✅ 🧪 mac-wk1 ✅ 🛠 gtk
✅ 🧪 mac-wk2 ✅ 🧪 gtk-wk2
✅ 🧪 services ✅ 🛠 vision ✅ 🧪 mac-AS-debug-wk2 ✅ 🧪 api-gtk
✅ 🛠 vision-sim ✅ 🧪 mac-wk2-stress ✅ 🛠 playstation
✅ 🧪 vision-wk2 ✅ 🧪 mac-intel-wk2
✅ 🛠 tv
✅ 🛠 tv-sim
✅ 🛠 watch
✅ 🛠 watch-sim

https://bugs.webkit.org/show_bug.cgi?id=314459
rdar://176618242

Reviewed by NOBODY (OOPS!).

`run-webkit-tests fast/dom/ChildNode-replaceWith.html` wall clock drops from ~5s to ~2.8s.
WebKitTestRunner itself still takes ~1s; the rest is setup overhead that this change trims.

Each of these improvements was independently measurable:
- LayoutTestHelper's ~440ms color-profile install now overlaps with test discovery instead
of blocking _set_up_run.
- configuration_for_upload (which pulls in requests, ~80ms) runs only when --report is passed.
- webkit-build-directory perl calls: 4 → 1 via a class-level cache and a lazy --target
default (~500ms).
- WebKitTestRunner warmup is skipped when child_processes == 1.
- Several production import paths no longer pull in unittest.mock, webkitcorepy.mocks
submodules, the whole requests stack, or the embedded-port chain on a Mac run.
- Misc: cached inner finder, precompiled regex, lazy port-factory table.

* Tools/Scripts/webkitpy/layout_tests/models/test_expectations.py:
(TestExpectationParser._tokenize_line):
Collapse whitespace with `" ".join(s.split())` instead of a regex, and precompile _BUG_TOKEN_RE.

* Tools/Scripts/webkitpy/port/base.py:
(Port.start_helper_async):
(Port.wait_for_helper_ready):
New no-op defaults; MacPort overrides. Splitting start_helper into a kick-off + join lets
the caller overlap the helper with other work.
(Port.stop_helper):
Reset _helper_start_attempted so a later run isn't mistaken for a failed one.

* Tools/Scripts/webkitpy/port/config.py:
(Config._build_directories):
(Config._clear_cache_for_testing):
(Config.build_directory):
Cache is class-level now (invariant within a process and sharable across Config instances)
and keyed on (port_implementation, configuration, for_host).

* Tools/Scripts/webkitpy/port/base_unittest.py:
* Tools/Scripts/webkitpy/port/config_unittest.py:
(PortTest.setUp):
(ConfigTest.setUp):
Clear the class cache between tests and between phases within test_build_directory.

* Tools/Scripts/webkitpy/port/darwin.py:
(DarwinPort.CURRENT_VERSION):
Import from version_name_map instead of literal Version(26), to avoid duplication.

* Tools/Scripts/webkitpy/port/factory.py:
(_lazy_port):
(PortFactory.PORT_CLASSES):
(PortFactory.get):
Replace the eager loop over all 14 port modules with a table mapping port-name prefix → CallByNeed.
Only the matched port is imported.
(configuration_options):
--target now defaults to None. Port.__init__ already calls set_option_default('configuration',
self.default_configuration()) after parsing, so the old eager Config.default_configuration()
call at option-definition time (one perl invocation every run) is redundant.

* Tools/Scripts/webkitpy/port/factory_unittest.py:
(FactoryTest.test_port_classes_table_consistency):
Assert each prefix in PORT_CLASSES matches the imported class's port_name so the duplication
can't silently drift.

* Tools/Scripts/webkitpy/port/mac.py:
(MacPort.setup_test_run):
Skip the warmup invocation when child_processes == 1.
(MacPort.start_helper):
(MacPort.start_helper_async):
(MacPort.wait_for_helper_ready):
Split into non-blocking popen + blocking readline. Track _ready on the Popen handle and
_helper_start_attempted on the class so wait returns False for a helper that failed to spawn.
@aproskuryakov aproskuryakov self-assigned this May 9, 2026
@aproskuryakov aproskuryakov added the Tools / Tests Tools in the Tools directory, build issues, test infrastructure, and bugs in test cases label May 9, 2026
@aproskuryakov aproskuryakov requested review from emw-apple and heycam May 9, 2026 03:27
@rr-codes
Copy link
Copy Markdown
Contributor

rr-codes commented May 9, 2026

I tried doing a similar thing for api-tests here, which ended up getting reverted, but maybe there's some parts of that change which might be helpful to roll into this change maybe? #60911

@aproskuryakov
Copy link
Copy Markdown
Contributor Author

I feel like this is already close to being too big, and not sure if any of these would be a speedup for this hero case. Worth looking into separately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Tools / Tests Tools in the Tools directory, build issues, test infrastructure, and bugs in test cases

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants