Do not report test-runner abortions and per-test timeouts as Server died

[leshikus]

Motivation

Follow-up to #105643, which folded every mid-flight abort exit code into a
single ABORTED_RUN_EXIT_CODES set and attached a synthetic Server died
leaf to all of them. That conflated unrelated conditions and produced
hundreds of spurious Server died rows from ordinary job-level timeouts and
flaky/targeted checks, where the server was healthy and shut down cleanly
(reported by @Algunenano; see e.g. the targeted run on #105019 and the flaky
check on #104965, and the alternative #106228).

Working through where each "timeout" actually originates showed there is no
single signal to fix - the cases are genuinely different and should be
reported differently.

Changes

tests/clickhouse-test — the per-test SIGALRM backstop (the watchdog for
completely frozen tests, armed at int(args.timeout * 1.1) + 60) used to
call stop_tests from inside the signal handler. That killpg SIGTERMs the
runner's own process group, so the run exits 143, which the job side then
misreports as an aborted run / Server died. The timed-out test is now
recorded as a normal per-test TestResult(FAIL, FailureReason.TIMEOUT)
through the usual process_result path - exactly like the existing
query-level and socket.timeout paths already do - so it surfaces as a plain
[ FAIL ] line with Reason: Timeout!. No exit-code-based reclassification
is needed on the job side. The test's client runs in its own session
(start_new_session=True), so it is still reaped by --cleanup, not by the
removed killpg.

ci/jobs/scripts/functional_tests_results.py:

• STOP_TESTING_EXIT_CODE (the in-band signal that the server actually died
  or the hung-check tripped) keeps the Server died leaf and the demotion of
  partial per-test results.
• An external kill - 143 / 137 / -15 / -9 (job-level timeout, runner
  shutdown, the worker→parent SIGTERM feedback loop) - is now reported as
  ERROR with a clickhouse-test leaf, not Server died. It is the same
  "the runner did not finish" condition as the existing not s.success_finish
  branch, and the per-test results that completed before the kill are kept
  authoritative (no UNKNOWN demotion). As ERROR it also routes the
  bugfix-validation inverter down its "preserve, do not invert" path: an abort
  is reported as inconclusive rather than silently claimed as "bug reproduced"
  (a kill does not prove the bug reproduced).
• Removed the dead order keys SERVER_DIED / Timeout / BROKEN (no leaf
  ever has those statuses) and added the real ERROR status, which was
  missing and silently sorted to the front.

Caveat

Dropping stop_tests from the SIGALRM handler means a single frozen test no
longer aborts the whole run via killpg - it is a FAIL and the run
continues, bounded by --max-failures-chain. This is strictly more resilient
than before (previously one frozen test killed the entire run with 143). The
outer Shell.run safety-net timeout still backstops a genuinely stuck run.

Changelog category (leave one):

• CI Fix or Improvement (changelog entry is not required)

[clickhouse-gh]

Workflow [PR], commit [ec8e646]

Summary: ✅

---

AI Review

Summary

This PR changes functional-test result classification so STOP_TESTING_EXIT_CODE remains a real Server died, external runner kills become an inconclusive ERROR, and a per-test SIGALRM timeout should become a normal timeout failure. I found one correctness issue in the new per-test timeout path, so this should be fixed before merge.

Findings

⚠️ Majors

• [tests/clickhouse-test:4457] The new except TimeoutError branch does not catch the actual per-test alarm in the usual path. timeout_handler raises Python's built-in TimeoutError while execution is inside test_case.run, but TestCase.run catches it with the broad except Exception at tests/clickhouse-test:3636 and returns TestStatus.UNKNOWN / FailureReason.INTERNAL_ERROR. A frozen test is therefore reported as [ UNKNOWN ] with Reason: Test internal error, not the promised normal [ FAIL ] with Reason: Timeout!. Move timeout handling before the broad exception path, or construct the timeout TestResult inside TestCase.run while still killing/reaping the still-running client process group.

Tests

⚠️ Add a focused regression test that triggers the per-test SIGALRM timeout path and verifies the emitted result is [ FAIL ] with Reason: Timeout!, not [ UNKNOWN ].

Missing context / blind spots

⚠️ The current Praktika S3 report for this PR has no test results yet (Total: 0), so I could not validate this against a completed CI run. A completed CI run with the new tests would close this gap.

Final Verdict

Status: ⚠️ Request changes

Minimum required actions: fix the per-test alarm handling so it reports the intended timeout failure without leaking the client process group, and cover that path with a focused regression test.

[clickhouse-gh]

This branch does not catch the actual per-test alarm in the usual path. The signal handler raises Python's built-in TimeoutError while execution is inside test_case.run, but TestCase.run catches it with the broad except Exception at tests/clickhouse-test:3636 and returns TestStatus.UNKNOWN/FailureReason.INTERNAL_ERROR; control never reaches this except TimeoutError. As a result a frozen test is reported as [ UNKNOWN ] with Reason: Test internal error, not as the promised normal [ FAIL ] with Reason: Timeout!.

Please handle the alarm before the generic exception path, or move the timeout result construction into TestCase.run while still killing/reaping the still-running client process group. A focused test that triggers this alarm path would prevent this from regressing.