[Tools] linux_get_crash_log: Revamp crash log generation for Linux layout tests

[clopez]

aaa99ed

[Tools] linux_get_crash_log: Revamp crash log generation for Linux layout tests
https://bugs.webkit.org/show_bug.cgi?id=315073

Reviewed by Nikolas Zimmermann.

Rewrite the Linux crash log tooling to be more reliable and informative.
The coredump location is now determined by reading /proc/sys/kernel/core_pattern
directly instead of relying on a hardcoded WEBKIT_CORE_DUMPS_DIRECTORY environment
variable. Both raw coredump files (abspath method) and systemd-coredump via
coredumpctl are supported, with automatic detection of which one is configured.

Coredump files are now matched to the crashing process by PID when possible,
using the %p format specifier from core_pattern. A fallback matching by program
name and timestamp is available but disabled by default since it is unreliable
when several test workers run in parallel. This can be tuned via environment
variables documented in the help message shown on missing coredumps.

A new inotify-based background thread captures thread names from /proc/<pid>/task
the moment the kernel creates the coredump file, before the process fully exits.
This information is included in the crash log to help identify the crashing thread.
The thread runs as a singleton across concurrent test runner instances using a
lockfile to avoid races when writing the thread name data.

GDB backtraces now use "bt full" to include local variables, and optionally wrap
the GDB invocation with GNU time to report memory and CPU usage. Parallel GDB
execution can be rate-limited via WEBKIT_CRASHLOG_GDB_CONCURRENT_EXECUTION_LIMIT
to avoid memory exhaustion during runs with many simultaneous crashes. Running
inside a PID namespace (e.g. containers) is detected and warned about, since it
prevents reliable coredump attribution when using the coredumpctl method.

* Tools/Scripts/webkitpy/__init__.py:
* Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py:
(run):
* Tools/Scripts/webkitpy/port/glib.py:
(GLibPort.setup_environ_for_server):
* Tools/Scripts/webkitpy/port/linux_get_crash_log.py:
(StrEnum):
(StrEnum.__str__):
(PrefixAdapter):
(PrefixAdapter.process):
(PrefixAdapter.log):
(LockFile):
(LockFile.__init__):
(LockFile.acquire):
(LockFile._try_claim):
(LockFile._acquire_single):
(LockFile._acquire_multi):
(LockFile.release):
(LockFile._log_waiter):
(LockFile.__enter__):
(LockFile.__exit__):
(CoreDumpMethod):
(CrashLogEnvVars):
(CrashLogUtils):
(CrashLogUtils.get_gdb_concurrent_execution_limit):
(CrashLogUtils._get_gdb_lock_configuration):
(CrashLogUtils.get_gdb_lock):
(CrashLogUtils.get_thread_info_file_name):
(CrashLogUtils.get_thread_data_dir):
(CrashLogUtils.get_temp_dir_for_coredumpctl_dumps):
(CrashLogUtils.get_temp_dir_for_coredumpctl_dumps.is_tmpfs):
(CrashLogUtils.get_temp_dir_for_coredumpctl_dumps.check_and_return_subdir):
(CrashLogUtils.allow_unreliable_fallback_to_latest_coredump):
(CrashLogUtils.expand_core_pattern_and_detect_variable_specifiers):
(CrashLogUtils.determine_coredump_method_and_dir):
(CrashLogUtils.core_pattern_to_regex):
(CrashLogUtils.are_coredumps_enabled):
(CrashLogUtils.are_coredumps_enabled_and_unlimited):
(CrashLogUtils.human_readable_size):
(CrashLogUtils.human_readable_size.fmt):
(CrashLogUtils.make_temp_path):
(CrashLogUtils.in_host_pid_namespace):
(CrashLogUtils.core_pattern_has_pid_format_string):
(ThreadNamesCrashLogCapturer):
(ThreadNamesCrashLogCapturer.__init__):
(ThreadNamesCrashLogCapturer._get_regex_for_core_pattern_pid_match):
(ThreadNamesCrashLogCapturer.watch_coredump_dir):
(ThreadNamesCrashLogCapturer.read_thread_info):
(ThreadNamesCrashLogCapturer.handle_coredump):
(GDBCrashLogStartupHandler):
(GDBCrashLogStartupHandler.__init__):
(GDBCrashLogStartupHandler._maybe_remove_file_if_old):
(GDBCrashLogStartupHandler._is_debuginfod_server_available):
(GDBCrashLogStartupHandler._check_debuginfod_servers):
(GDBCrashLogStartupHandler.clean_old_coredumps):
(GDBCrashLogStartupHandler.clean_old_thread_info_files):
(GDBCrashLogGenerator):
(GDBCrashLogGenerator.__init__):
(GDBCrashLogGenerator._get_gdb_output):
(GDBCrashLogGenerator._coredump_pattern_has_pid_format_string):
(GDBCrashLogGenerator._pid_is_valid):
(GDBCrashLogGenerator._build_warning_msg):
(GDBCrashLogGenerator._pick_most_recent):
(GDBCrashLogGenerator._get_coredump_path_with_core_pattern_method):
(GDBCrashLogGenerator._coredumpctl_dump_core_for_pid):
(GDBCrashLogGenerator._get_coredump_path_with_coredumpctl_method):
(GDBCrashLogGenerator._get_coredump_path):
(GDBCrashLogGenerator._get_help_message):
(GDBCrashLogGenerator._get_thread_info_for_effective_pid):
(GDBCrashLogGenerator._pid_representation):
(GDBCrashLogGenerator._filter_with_cppfilt):
(GDBCrashLogGenerator._underscore_header):
(GDBCrashLogGenerator.generate_crash_log):
(GDBCrashLogGenerator._get_trace_from_systemd): Deleted.
(GDBCrashLogGenerator.generate_crash_log.match_filename): Deleted.
* Tools/Scripts/webkitpy/port/linux_get_crash_log_unittest.py:
(_make_generator):
(AreCoredumpsEnabledTest):
(AreCoredumpsEnabledTest.test_coredumpctl_always_enabled):
(AreCoredumpsEnabledTest.test_abspath_checks_rlimit):
(CorePatternToRegexTest):
(CorePatternToRegexTest.test_literal_string_passes_through):
(CorePatternToRegexTest.test_specifier_becomes_wildcard):
(CorePatternToRegexTest.test_named_pid_capture_group):
(CorePatternToRegexTest.test_pid_capture_only_when_requested):
(CorePatternToRegexTest.test_double_percent_is_literal_percent):
(CorePatternToRegexTest.test_trailing_lone_percent_is_dropped):
(CorePatternToRegexTest.test_invalid_specifier_is_dropped):
(CorePatternToRegexTest.test_regex_metacharacters_are_escaped):
(CorePatternToRegexTest.test_full_path_with_multiple_specifiers):
(ExpandCorePatternTest):
(ExpandCorePatternTest.test_no_specifiers):
(ExpandCorePatternTest.test_fixed_specifier_uid_is_expanded):
(ExpandCorePatternTest.test_variable_specifier_pid_is_detected):
(ExpandCorePatternTest.test_double_percent_preserved_as_literal):
(ExpandCorePatternTest.test_trailing_lone_percent_is_dropped):
(HumanReadableSizeTest):
(HumanReadableSizeTest._file_of_size):
(HumanReadableSizeTest.test_small_file_reports_bytes):
(HumanReadableSizeTest.test_few_kilobyte_file_reports_kb):
(HumanReadableSizeTest.test_empty_file):
(MakeTempPathTest):
(MakeTempPathTest.test_creates_a_real_empty_file):
(MakeTempPathTest.test_respects_suffix):
(MakeTempPathTest.test_respects_dir):
(MakeTempPathTest.test_paths_are_unique):
(LockFileTest):
(LockFileTest.setUp):
(LockFileTest.test_acquire_creates_and_release_removes_lockfile):
(LockFileTest.test_context_manager):
(LockFileTest.test_release_when_not_acquired_is_noop):
(LockFileTest.test_lockfile_contains_pid):
(LockFileTest.test_single_slot_serializes_two_threads):
(LockFileTest.test_single_slot_serializes_two_threads.worker):
(LockFileTest.test_multi_slot_allows_concurrent_holders):
(LockFileTest.test_get_gdb_lock_returns_distinct_lock_instances):
(GDBCrashLogGeneratorHelpersTest):
(GDBCrashLogGeneratorHelpersTest.setUp):
(GDBCrashLogGeneratorHelpersTest.test_pid_representation_known):
(GDBCrashLogGeneratorHelpersTest.test_pid_representation_none):
(GDBCrashLogGeneratorHelpersTest.test_pid_is_valid_when_numeric):
(GDBCrashLogGeneratorHelpersTest.test_pid_is_invalid_when_none):
(GDBCrashLogGeneratorHelpersTest.test_pid_is_invalid_when_non_numeric_string):
(GDBCrashLogGeneratorHelpersTest.test_pattern_with_pid_specifier):
(GDBCrashLogGeneratorHelpersTest.test_pattern_without_pid_specifier):
(GDBCrashLogGeneratorHelpersTest.test_underscore_header_basic):
(GDBCrashLogGeneratorHelpersTest.test_underscore_header_first_omits_leading_newlines):
(BuildWarningMsgTest):
(BuildWarningMsgTest.setUp):
(BuildWarningMsgTest.test_pid_found_returns_none):
(BuildWarningMsgTest.test_name_found_mentions_program_name):
(BuildWarningMsgTest.test_last_found_warns_high_risk):
(BuildWarningMsgTest.test_nothing_found_default_path_suggests_env_var):
(BuildWarningMsgTest.test_no_flag_set_raises):
(PickMostRecentTest):
(PickMostRecentTest.setUp):
(PickMostRecentTest._touch):
(PickMostRecentTest.test_empty_candidates_returns_none):
(PickMostRecentTest.test_no_pattern_match_returns_none):
(PickMostRecentTest.test_picks_newest_and_extracts_pid):
(PickMostRecentTest.test_warning_kwargs_passed_through):
(GenerateCrashLogTest):
(GenerateCrashLogTest.test_no_coredump_emits_help_message_and_sections):
(GenerateCrashLogTest.test_stderr_section_skipped_when_no_stderr):
(DetermineCoredumpMethodAndDirTest):
(DetermineCoredumpMethodAndDirTest._run_with_core_pattern):
(DetermineCoredumpMethodAndDirTest.test_abspath_method_simple):
(DetermineCoredumpMethodAndDirTest.test_abspath_with_fixed_specifier_in_dir):
(DetermineCoredumpMethodAndDirTest.test_abspath_with_variable_specifier_in_dir):
(DetermineCoredumpMethodAndDirTest.test_systemd_pipe_with_coredumpctl_present):
(DetermineCoredumpMethodAndDirTest.test_systemd_pipe_without_coredumpctl_program):
(DetermineCoredumpMethodAndDirTest.test_pipe_to_non_systemd_program):
(DetermineCoredumpMethodAndDirTest.test_unknown_when_pattern_is_relative):
(GetTempDirForCoredumpctlDumpsTest):
(GetTempDirForCoredumpctlDumpsTest.setUp):
(GetTempDirForCoredumpctlDumpsTest._run):
(GetTempDirForCoredumpctlDumpsTest._run.fake_open):
(GetTempDirForCoredumpctlDumpsTest.test_picks_tmp_when_only_tmp_writable):
(GetTempDirForCoredumpctlDumpsTest.test_picks_var_tmp_when_only_var_tmp_writable):
(GetTempDirForCoredumpctlDumpsTest.test_picks_tmp_when_free_space_is_similar):
(GetTempDirForCoredumpctlDumpsTest.test_picks_var_tmp_when_substantially_more_free):
(GetTempDirForCoredumpctlDumpsTest.test_skips_tmpfs_mount):
(GetTempDirForCoredumpctlDumpsTest.test_falls_back_to_tmp_when_nothing_usable):
(ThreadNamesCrashLogCapturerTest):
(ThreadNamesCrashLogCapturerTest.setUp):
(ThreadNamesCrashLogCapturerTest._read_output):
(ThreadNamesCrashLogCapturerTest.test_capturer_is_disabled_when_pattern_only_contains_literal_percent_p):
(ThreadNamesCrashLogCapturerTest.test_repeated_pid_specifiers_reuse_the_same_pid_capture):
(ThreadNamesCrashLogCapturerTest.test_writes_thread_table_when_threads_present):
(ThreadNamesCrashLogCapturerTest.test_writes_error_when_process_already_gone):
(ThreadNamesCrashLogCapturerTest.test_writes_warning_when_no_threads_found):
(ThreadNamesCrashLogCapturerTest.test_full_path_through_fake_proc_task):
(ThreadNamesCrashLogCapturerTest.test_full_path_through_fake_proc_task.fake_listdir):
(ThreadNamesCrashLogCapturerTest.test_full_path_through_fake_proc_task.fake_open):
(GetCoredumpPathTest):
(GetCoredumpPathTest.setUp):
(GetCoredumpPathTest._touch):
(GetCoredumpPathTest.test_matches_by_pid_when_available):
(GetCoredumpPathTest.test_matches_by_pid_when_pattern_contains_literal_percent_p):
(GetCoredumpPathTest.test_does_not_fall_back_to_name_when_pid_match_is_possible_and_fallback_is_disabled):
(GetCoredumpPathTest.test_falls_back_to_name_when_pid_match_is_possible_and_fallback_is_enabled):
(GetCoredumpPathTest.test_falls_back_to_name_when_pid_is_invalid_and_fallback_is_disabled):
(GetCoredumpPathTest.test_falls_back_to_name_when_core_pattern_cannot_match_pid_and_fallback_is_disabled):
(GetCoredumpPathTest.test_pid_match_on_second_attempt_wins_over_existing_name_match):
(GetCoredumpPathTest.test_pid_match_on_second_attempt_wins_over_existing_name_match.create_pid_match):
(GetCoredumpPathTest.test_pattern_with_literal_percent_p_has_no_pid_specifier):
(GetCoredumpPathTest.test_pid_lookup_ignores_filename_with_extra_suffix):
(GetCoredumpPathTest.test_returns_none_without_unreliable_fallback):
(GetCoredumpPathTest.test_filters_by_newer_than):
(GetCoredumpPathWithCoredumpctlTest):
(GetCoredumpPathWithCoredumpctlTest.setUp):
(GetCoredumpPathWithCoredumpctlTest._entries_json):
(GetCoredumpPathWithCoredumpctlTest._entry):
(GetCoredumpPathWithCoredumpctlTest._dump_core_side_effect):
(GetCoredumpPathWithCoredumpctlTest.test_matches_by_pid_when_pid_is_valid_and_in_host_namespace):
(GetCoredumpPathWithCoredumpctlTest.test_does_not_fall_back_to_name_when_pid_match_is_possible_and_fallback_is_disabled):
(GetCoredumpPathWithCoredumpctlTest.test_falls_back_to_name_when_pid_match_is_possible_but_fallback_is_enabled):
(GetCoredumpPathWithCoredumpctlTest.test_falls_back_to_name_without_env_var_when_running_in_pid_namespace):
(GetCoredumpPathWithCoredumpctlTest.test_falls_back_to_name_without_env_var_when_pid_is_invalid):
(GetCoredumpPathWithCoredumpctlTest.test_falls_back_to_latest_only_when_unreliable_fallback_is_enabled):
(GDBCrashLogGeneratorTest): Deleted.
(GDBCrashLogGeneratorTest.test_generate_crash_log): Deleted.
* Tools/glib/dependencies/apt:
* Tools/glib/dependencies/dnf:
* Tools/glib/dependencies/pacman:

Canonical link: https://commits.webkit.org/313530@main

9e21c38

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	🛠 ios	🛠 mac	🛠 wpe	🛠 win
✅ 🧪 bindings	🛠 ios-sim	🛠 mac-AS-debug	🧪 wpe-wk2	🧪 win-tests
✅ 🧪 webkitperl	🧪 ios-wk2	🧪 api-mac	🧪 api-wpe
✅ 🧪 webkitpy	🧪 ios-wk2-wpt	🧪 api-mac-debug	🛠 gtk3-libwebrtc
	🧪 api-ios	🧪 mac-wk1	🛠 gtk
		🧪 mac-wk2	🧪 gtk-wk2
✅ 🧪 services	🛠 vision	🧪 mac-AS-debug-wk2	🧪 api-gtk
	🛠 vision-sim	🧪 mac-wk2-stress	🛠 playstation
✅ 🛠 🧪 unsafe-merge	🧪 vision-wk2	🧪 mac-intel-wk2
	🛠 tv
	🛠 tv-sim	🧪 mac-site-isolation
	🛠 watch
	🛠 watch-sim

[webkit-early-warning-system]

EWS run on previous version of this PR (hash f7dfc53)

Details

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 win
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	✅ 🧪 wpe-wk2	❌ 🧪 win-tests
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	✅ 🧪 api-wpe
❌ 🧪 webkitpy	🧪 ios-wk2-wpt	✅ 🧪 api-mac-debug	✅ 🛠 gtk3-libwebrtc
	✅ 🧪 api-ios	✅ 🧪 mac-wk1	✅ 🛠 gtk
		✅ 🧪 mac-wk2	✅ 🧪 gtk-wk2
✅ 🧪 services	✅ 🛠 vision	✅ 🧪 mac-AS-debug-wk2	✅ 🧪 api-gtk
	✅ 🛠 vision-sim	✅ 🧪 mac-wk2-stress	✅ 🛠 playstation
	✅ 🧪 vision-wk2	✅ 🧪 mac-intel-wk2
	✅ 🛠 tv
	✅ 🛠 tv-sim	✅ 🧪 mac-site-isolation
	✅ 🛠 watch
	✅ 🛠 watch-sim

[nikolaszimmermann]

Looks great, r=me.

[webkit-early-warning-system]

EWS run on previous version of this PR (hash 31d3562)

Details

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 win
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	✅ 🧪 wpe-wk2	❌ 🧪 win-tests
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	✅ 🧪 api-wpe
✅ 🧪 webkitpy	🧪 ios-wk2-wpt	✅ 🧪 api-mac-debug	✅ 🛠 gtk3-libwebrtc
	✅ 🧪 api-ios	✅ 🧪 mac-wk1	✅ 🛠 gtk
		✅ 🧪 mac-wk2	✅ 🧪 gtk-wk2
✅ 🧪 services	✅ 🛠 vision	✅ 🧪 mac-AS-debug-wk2	✅ 🧪 api-gtk
	✅ 🛠 vision-sim	✅ 🧪 mac-wk2-stress	✅ 🛠 playstation
	✅ 🧪 vision-wk2	✅ 🧪 mac-intel-wk2
	✅ 🛠 tv
	✅ 🛠 tv-sim	✅ 🧪 mac-site-isolation
	✅ 🛠 watch
	✅ 🛠 watch-sim

[webkit-early-warning-system]

EWS run on current version of this PR (hash 9e21c38)

Details

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	🛠 ios	🛠 mac	🛠 wpe	🛠 win
✅ 🧪 bindings	🛠 ios-sim	🛠 mac-AS-debug	🧪 wpe-wk2	🧪 win-tests
✅ 🧪 webkitperl	🧪 ios-wk2	🧪 api-mac	🧪 api-wpe
✅ 🧪 webkitpy	🧪 ios-wk2-wpt	🧪 api-mac-debug	🛠 gtk3-libwebrtc
	🧪 api-ios	🧪 mac-wk1	🛠 gtk
		🧪 mac-wk2	🧪 gtk-wk2
✅ 🧪 services	🛠 vision	🧪 mac-AS-debug-wk2	🧪 api-gtk
	🛠 vision-sim	🧪 mac-wk2-stress	🛠 playstation
✅ 🛠 🧪 unsafe-merge	🧪 vision-wk2	🧪 mac-intel-wk2
	🛠 tv
	🛠 tv-sim	🧪 mac-site-isolation
	🛠 watch
	🛠 watch-sim

[webkit-commit-queue]

Committed 313530@main (aaa99ed): https://commits.webkit.org/313530@main

Reviewed commits have been landed. Closing PR #65162 and removing active labels.