Skip to content

[lldb][Windows] Dump thread stacks before lit's timeout kills a test - #213239

Open
charles-zablit wants to merge 2 commits into
llvm:mainfrom
charles-zablit:cz/lldb/windows/dotest-hang-stackdump
Open

[lldb][Windows] Dump thread stacks before lit's timeout kills a test#213239
charles-zablit wants to merge 2 commits into
llvm:mainfrom
charles-zablit:cz/lldb/windows/dotest-hang-stackdump

Conversation

@charles-zablit

Copy link
Copy Markdown
Contributor

On Windows, when a test hits a timeout, we currently don't get a stacktrace. dotest solves this on POSIX: it registers a SIGTERM handler, so a killed test prints its stack. The handler does not run on Windows however because faulthandler.register doesn't exist there, and lit terminates the process instead of signalling it (no signals on Windows).

This patch adds faulthandler.dump_traceback_later() with a default timeout value of 300s. That's longer than the longest test in CI (~150s) and does not kill the test. It simply dumps the stacktrace at a point where the test is very likely stuck.

Before

TIMEOUT: lldb-api :: types/TestFloatTypesExpr.py (2698 of 2698)
******************** TEST 'lldb-api :: types/TestFloatTypesExpr.py' FAILED ********************
Exit Code: 15
Timeout: Reached timeout of 900 seconds

There is no stacktrace.

After

Timeout (0:00:20)!
Thread 0x00012fe8 (most recent call first):
  File ".../types/TestFloatTypesExpr.py", line 22 in test_float_type
  File ".../lldbsuite/test/lldbtest.py", line 2097 in test_method
  File ".../unittest/case.py", line 549 in _callTestMethod
  ...
  File ".../lldbsuite/test/dotest.py", line 1212 in run_suite

@llvmorg-github-actions

Copy link
Copy Markdown

@llvm/pr-subscribers-lldb

Author: Charles Zablit (charles-zablit)

Changes

On Windows, when a test hits a timeout, we currently don't get a stacktrace. dotest solves this on POSIX: it registers a SIGTERM handler, so a killed test prints its stack. The handler does not run on Windows however because faulthandler.register doesn't exist there, and lit terminates the process instead of signalling it (no signals on Windows).

This patch adds faulthandler.dump_traceback_later() with a default timeout value of 300s. That's longer than the longest test in CI (~150s) and does not kill the test. It simply dumps the stacktrace at a point where the test is very likely stuck.

Before

TIMEOUT: lldb-api :: types/TestFloatTypesExpr.py (2698 of 2698)
******************** TEST 'lldb-api :: types/TestFloatTypesExpr.py' FAILED ********************
Exit Code: 15
Timeout: Reached timeout of 900 seconds

There is no stacktrace.

After

Timeout (0:00:20)!
Thread 0x00012fe8 (most recent call first):
  File ".../types/TestFloatTypesExpr.py", line 22 in test_float_type
  File ".../lldbsuite/test/lldbtest.py", line 2097 in test_method
  File ".../unittest/case.py", line 549 in _callTestMethod
  ...
  File ".../lldbsuite/test/dotest.py", line 1212 in run_suite

Full diff: https://github.com/llvm/llvm-project/pull/213239.diff

1 Files Affected:

  • (modified) lldb/packages/Python/lldbsuite/test/dotest.py (+13)
diff --git a/lldb/packages/Python/lldbsuite/test/dotest.py b/lldb/packages/Python/lldbsuite/test/dotest.py
index 604fb98b1e2e5..3c966dceb96c7 100644
--- a/lldb/packages/Python/lldbsuite/test/dotest.py
+++ b/lldb/packages/Python/lldbsuite/test/dotest.py
@@ -504,6 +504,19 @@ def registerFaulthandler():
     if getattr(faulthandler, "register", None):
         faulthandler.register(signal.SIGTERM, chain=True)
 
+    if sys.platform != "win32":
+        return
+
+    # lit kills a hung test with TerminateProcess on Windows, so the SIGTERM
+    # handler above never runs and a timeout is reported with no indication of
+    # where it hung. Dump every thread's stack while the process is still alive.
+    try:
+        secs = float(os.environ.get("LLDB_TEST_STACK_DUMP_SECS", 300))
+    except ValueError:
+        secs = 300
+    if secs > 0:
+        faulthandler.dump_traceback_later(secs, exit=False)
+
 
 def setupSysPath():
     """

Comment thread lldb/packages/Python/lldbsuite/test/dotest.py Outdated
try:
secs = float(os.environ.get("LLDB_TEST_STACK_DUMP_SECS", 300))
except ValueError:
secs = 300

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should not hardcode any timeouts let alone in different places. Would it be possible to derive this from the per-test timeout setting?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done through a new argument in the dotest invocation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants