//target/earlgrey/tests/uart:uart_qemu_test intermittently hangs in CI and dies on the 120s bazel test timeout. It has hit both main and an unrelated PR within two days.
Occurrences — all failures on openprot-ci-runner-4
| Run |
Branch / event |
Runner |
Test job result |
| 30110746702 |
push to main (2026-07-24) |
openprot-ci-runner-4 |
uart_qemu_test FAILED in 124.2s (timeout) |
| 30116154309 |
PR #368 (2026-07-24) |
openprot-ci-runner-4 |
failure (Test job took 2h23m) |
| 30160896818 |
PR #368 (2026-07-25) |
openprot-ci-runner-4 |
uart_qemu_test FAILED in 122.4s (timeout) |
| 30116105172 |
PR #368 (2026-07-24) |
openprot-ci-runner-3 |
passed |
| 30144173373 |
push to main (2026-07-25) |
openprot-ci-runner |
passed |
| 30067435354 |
push to main (2026-07-24) |
openprot-ci-runner-5 |
passed (7m) |
| 29996400464 |
push to main (2026-07-23) |
openprot-ci-runner-2 |
passed |
Every recent failure landed on openprot-ci-runner-4; every pass on a different runner. Two more signs the host is unhealthy: the Test job on runner-4 took 2h23m in run 30116154309 (vs 7m on runner-5 for a comparable main run), and the post-job cleanup in run 30160896818 had to terminate orphan processes (two leftover java pids).
The PR (#368) only touches services/fwmanager/**, nothing under target/earlgrey, and its run on runner-3 passed — so this is not change-related. The other three earlgrey QEMU tests (gpio_qemu_test, ipc_runner_qemu_test, threads_runner_qemu_test) pass in 3–15s even in the failing runs.
Symptom
The test boots QEMU and the kernel fine, then goes silent during test process/thread allocation, before any UART test output:
Run 30160896818 (PR #368):
[INF] Welcome to the first thread, continuing bootstrap
[INF] Allocating non-privileged process 'test_uart'
[INF] Allocating non-privileged thread 'uart_thread'
ERROR: test timed out after 120s
Run 30110746702 (main):
[INF] Welcome to the first thread, continuing bootstrap
[INF] Allocating non-privileged process 'test_uart'
[INF] Allocating non-privileged thread 'uart_thread'
[INF] Allocating non-privileged process 'test_uart_listener'
ERROR: test timed out after 120s
The hang point varies by one line between the two occurrences.
Likely cause
The runner correlation points at the openprot-ci-runner-4 host rather than (or in addition to) a code race: the riscv32 guest runs under pure TCG emulation, so a heavily loaded or degraded host starves the guest and the 120s wall-clock timeout expires while the guest sits mid-startup. uart_qemu_test may simply be the longest-running/most timing-sensitive of the four QEMU tests, so it's the one that trips first.
Suggest checking openprot-ci-runner-4 for leaked QEMU/bazel/java processes or other load, and/or taking it out of the pool until it's healthy. If failures also appear on other runners, then a genuine startup race in test process allocation is back on the table.
Notes
//target/earlgrey/tests/uart:uart_qemu_testintermittently hangs in CI and dies on the 120s bazel test timeout. It has hit bothmainand an unrelated PR within two days.Occurrences — all failures on
openprot-ci-runner-4main(2026-07-24)uart_qemu_testFAILED in 124.2s (timeout)uart_qemu_testFAILED in 122.4s (timeout)main(2026-07-25)main(2026-07-24)main(2026-07-23)Every recent failure landed on
openprot-ci-runner-4; every pass on a different runner. Two more signs the host is unhealthy: the Test job on runner-4 took 2h23m in run 30116154309 (vs 7m on runner-5 for a comparable main run), and the post-job cleanup in run 30160896818 had to terminate orphan processes (two leftover java pids).The PR (#368) only touches
services/fwmanager/**, nothing undertarget/earlgrey, and its run on runner-3 passed — so this is not change-related. The other three earlgrey QEMU tests (gpio_qemu_test,ipc_runner_qemu_test,threads_runner_qemu_test) pass in 3–15s even in the failing runs.Symptom
The test boots QEMU and the kernel fine, then goes silent during test process/thread allocation, before any UART test output:
Run 30160896818 (PR #368):
Run 30110746702 (main):
The hang point varies by one line between the two occurrences.
Likely cause
The runner correlation points at the
openprot-ci-runner-4host rather than (or in addition to) a code race: the riscv32 guest runs under pure TCG emulation, so a heavily loaded or degraded host starves the guest and the 120s wall-clock timeout expires while the guest sits mid-startup.uart_qemu_testmay simply be the longest-running/most timing-sensitive of the four QEMU tests, so it's the one that trips first.Suggest checking
openprot-ci-runner-4for leaked QEMU/bazel/java processes or other load, and/or taking it out of the pool until it's healthy. If failures also appear on other runners, then a genuine startup race in test process allocation is back on the table.Notes