feat: add --per-depth-timeout option for kontrol prove

[Stevengre]

Summary

Adds --per-depth-timeout SECONDS to kontrol prove. When set (>0), each prove attempt runs in a forked subprocess. The parent treats current_depth * per_depth_timeout as a stall window rather than a hard total budget: every window the parent polls the proof's on-disk subdir for file mtime changes; any change means the prove committed a step and the window resets for another round. Only when a full window passes with no progress does the parent SIGKILL the entire subprocess session (Python + KoreServer subprocess + parallel-frontier worker threads), halve current_depth (floor 1), and start the next attempt — which resumes from the disk-persisted KCFG state. Default 0 disables the wrapper.

Example: --max-depth 1000 --per-depth-timeout 10 starts with a 10000s stall window at depth=1000. As long as KCFG keeps growing within each 10000s window, the prove runs as long as it likes. If no node is committed for 10000s, kill → halve to depth=500 with a 5000s window → … → depth=1 with a 10s window.

Why stall window (vs. hard wall-clock cap)

A hard total budget kills proofs that are productively making progress just because the wall-clock crossed a threshold. The intent of progressive halving is to detect when execute_depth is too coarse to break out of a stuck step — exactly the case where no new nodes appear. Stall-window semantics matches the symptom directly: if the prove keeps writing to disk, leave it alone; if it stops writing, the next step is stuck and shrinking execute_depth is the right move.

Why subprocess + SIGKILL (vs. callback inside run_prover)

advance_proof's maintenance callback only fires after step_proof returns. If a single step takes minutes (deep symbolic execution, expensive SMT), the callback can't fire and a callback-based timeout can't interrupt. Subprocess + session-group SIGKILL gives precise wall-clock cutoff regardless of what the prover is doing internally — the kernel atomically reaps every relevant process and thread.

Implementation

• cli.py: new --per-depth-timeout flag.
• options.py: new ProveOptions.per_depth_timeout (default 0).
• prove.py:
  • When per_depth_timeout > 0, init_and_run_proof delegates to _init_and_run_proof_progressive, which loops over halving depths.
  • _run_attempt_under_timeout(test, attempt_max_depth, budget_s) runs one attempt:
    • mp.get_context('fork').Process so the child inherits everything via CoW (no pickling, no spawn cost).
    • Child calls os.setsid() to become a session leader, mutates its local fork-copy of options.max_depth and options.per_depth_timeout = 0 (preventing recursion), then re-enters init_and_run_proof. The new KoreServer it starts is in the same session.
    • Parent polls every budget_s via proc.join(timeout=budget_s). If child exited → break out and read the result via Pipe. If still alive → walk foundry.proofs_dir / test.id and compare max file mtime against the previous snapshot.
    • Marker changed → grant another budget_s window. Marker unchanged → os.killpg(os.getpgid(proc.pid), SIGKILL) reaps the whole session, return _ATTEMPT_TIMEOUT.

Test plan

• kontrol prove --help lists --per-depth-timeout.
• Default-off: existing test suite passes unchanged (no subprocess overhead, run_prover(...) is the unaltered code path).
• Stall detection: a contrived proof that hangs in a single long step_proof is killed within budget_s + poll_overhead; logs show depth=N attempt exhausted Ms budget; halving.
• No false positives: a productive proof making steady progress (any committed step inside each window) runs to completion without being killed.
• Hard kill cleanup: while a --per-depth-timeout 10 --max-depth 100 proof runs, pgrep kore-rpc-booster | wc -l returns to baseline within a second of the kill (kore-rpc subprocess reaped via session-group SIGKILL, not orphaned).
• Resumption: after a forced halving, the next attempt's KCFG starts from where the previous one left off (no full restart).
• workers > 1: each test's per-attempt subprocess is independent; no cross-contamination of progress signals (the marker walks proofs_dir / test.id, scoped per test).

Caveats

• Uses mp.get_context('fork'); requires POSIX (already a kontrol requirement for kore-rpc-booster).
• Each halving spawns a fresh KoreServer; per-attempt startup cost adds a few seconds.
• Atomicity: pyk's proof.write_proof_data() is assumed to write atomically (temp + rename). A SIGKILL during write could yield a partial state, but pyk's loader can re-fetch from logs.
• Polling cadence equals budget_s. Kill latency is therefore bounded by one extra window beyond actual stall onset — not millisecond-precise but adequate for budgets >= seconds.