Fix fuzzer status.tsv write aborting on "server_died: unbound variable"

[groeneai]

Changelog category (leave one):

• CI Fix or Improvement (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Not required (CI fix).

Description

ci/jobs/scripts/fuzzer/run-fuzzer.sh writes the run outcome to status.tsv
with echo -e "$server_died\t$server_exit_code\t$fuzzer_exit_code". Of the
three variables, server_died is the only one never initialized before use:
it is assigned solely inside the post-fuzz server-liveness loop, either
server_died=0 (SELECT 1 succeeds, then break) or server_died=1 (server
unreachable, then break).

Two branches of that loop, TOO_MANY_SIMULTANEOUS_QUERIES and
MEMORY_LIMIT_EXCEEDED, sleep and continue without assigning server_died
or breaking. When SELECT 1 keeps returning one of these for all 100
iterations (e.g. the tcp_handler_fail_connection_setup failpoint returns
MEMORY_LIMIT_EXCEEDED, or the server is under sustained memory pressure),
the loop exhausts with server_died still unset. Under set -u the
status.tsv write then aborts with server_died: unbound variable, so the
file is never created and ast_fuzzer_job.py:240 reports the job as ERROR:

FileNotFoundError: [Errno 2] No such file or directory: '.../ci/tmp/workspace/status.tsv'

even though the server shut down cleanly (signal 15, graceful teardown, no
crash, no sanitizer report, no core dump).

Fix: initialize server_died=0 before the loop, matching the existing
pre-loop defaults of its two siblings (fuzzer_exit_code=0,
server_exit_code=0). A truly dead server still sets server_died=1 in the
else branch; defaulting to 0 only covers loop exhaustion via the "alive
but busy" branches, which means the server survived.

Observed over the last 30 days across 112 distinct unrelated PRs and 17 master
runs, on all fuzzer variants (BuzzHouse amd_debug / arm_asan_ubsan /
amd_msan / amd_tsan and AST fuzzer).

Example CI report (BuzzHouse arm_asan_ubsan):
https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=107817&sha=7cae758e10f60706ccc862b31386f5447c67e324&name_0=PR&name_1=BuzzHouse%20%28arm_asan_ubsan%29

No related open issue found.

Version info

• Merged into: 26.6.1.1008

[groeneai]

Pre-PR validation gate

#	Question	Answer
a	Deterministic repro?	Yes. Minimal bash repro of fuzz()'s liveness loop under set -e -u -o pipefail: when every SELECT 1 returns MEMORY_LIMIT_EXCEEDED, the loop exhausts all retries without assigning server_died, and echo -e "$server_died\t..." > status.tsv aborts with exactly server_died: unbound variable, exit 1, no status.tsv written. This is byte-for-byte the CI failure (run-fuzzer.sh: line 325: server_died: unbound variable then FileNotFoundError: .../status.tsv at ast_fuzzer_job.py:240).
b	Root cause explained?	Yes. server_died is the only one of the three status.tsv fields never initialized before use; it is assigned only inside the loop (=0 on SELECT 1 success+break, =1 on the unreachable else+break). The TOO_MANY_SIMULTANEOUS_QUERIES and MEMORY_LIMIT_EXCEEDED branches sleep and continue without assigning it or breaking. If SELECT 1 keeps hitting those for all 100 iterations (e.g. tcp_handler_fail_connection_setup failpoint -> MEMORY_LIMIT_EXCEEDED), the loop exits with server_died unset, and set -u aborts the write. The arm_asan_ubsan job's server.log confirms a clean shutdown (signal 15, graceful teardown, no Fatal/ASAN/SIGSEGV/core) - no engine bug.
c	Fix matches root cause?	Yes. Initialize server_died=0 before the loop, mirroring the existing pre-loop defaults of its two siblings (fuzzer_exit_code=0, server_exit_code=0). Not a band-aid: the only path that previously left it unset is loop exhaustion via the "alive but busy" branches, which means the server survived, so 0 is the correct default. A genuinely dead server still sets server_died=1 in the else branch.
d	Test intent preserved / new tests added?	N/A in the unit-test sense (this is a CI harness shell script). The fix does not weaken any failure detection: server-death is still reported via the else branch; only the spurious abort on clean runs is removed.
e	Both directions demonstrated?	Yes. Without the fix: server_died: unbound variable, exit 1, no status.tsv. With server_died=0 added before the loop: writes status.tsv = 0\t0\t0, exit 0. bash -n passes on the modified script.
f	Fix is general, not a narrow patch?	Yes. There is a single run-fuzzer.sh and a single status.tsv write site; the same uninitialized-variable pattern does not exist for the other two fields (both pre-initialized). The fix is at the variable's origin, not a guard at the write site. Confirmed systemic: 112 distinct unrelated PRs + 17 master runs in 30 days across all fuzzer variants.

Server clean-shutdown evidence (arm_asan_ubsan, sha 7cae758): Received signal 15 -> KeeperDispatcher shut down -> disks destroyed -> "Destroyed global context" -> "Background threads finished in 721 ms" -> "Application: shutting down". sanitizer.log.615 contained only the benign ASan doesn't fully support makecontext/swapcontext startup warning.

Session id: cron:clickhouse-worker-slot-1:20260618-182400

[groeneai]

cc @maxknv @Algunenano — could you review? One-line CI harness fix: run-fuzzer.sh never initialized server_died before the post-fuzz liveness loop, so when every SELECT 1 returns MEMORY_LIMIT_EXCEEDED/TOO_MANY_SIMULTANEOUS_QUERIES (those branches don't assign it or break) the loop exhausts with it unset and set -u aborts the status.tsv write -> the job ERRORs with FileNotFoundError despite a clean server shutdown. Initializing server_died=0 before the loop fixes it (112 unrelated PRs + 17 master runs hit this in 30d).

[clickhouse-gh]

Workflow [PR], commit [7c6ba5b]

Summary: ✅

---

AI Review

Summary

This PR makes the fuzzer runner report status.tsv for the previously uninitialized server_died path, and the follow-up commit also makes the SHOW PROCESSLIST diagnostic best-effort under set -e. I did not find any remaining correctness or CI-harness issues in the current diff; the earlier bot-authored inline thread was fixed by the current code and has been resolved.

Final Verdict

Status: ✅ Approve