Skip to content

HistoryApp Phase 4 finalization is not interrupt-safe — interrupt after the wave block strands a non-terminal history_run #137

Description

@Sootopolis

Summary

HistoryApp.discoverClub's Phase 4 finalization (compute totalStatsHistoryRun.completelogSummary) sits outside the interrupt-safe block that wraps Phases 2–3. ZIO's .onInterrupt only guards the effect it wraps, so once the wave block yields and control moves into Phase 4, that finalizer has deregistered. An interrupt landing in the narrow Phase-4 window therefore runs neither finalizeInterrupted (the block's finalizer already passed) nor the rest of Phase 4 — leaving the history_run row in its non-terminal Phase-1 state with no compensating finalizer.

This is pre-existing (the block/Phase-4 split predates the recent reporting fix). Surfacing it now because moving logSummary into Phase 4 (the #136-follow-up reporting fix) widened the unguarded window by one effect — though HistoryRun.complete was already there and already unguarded.

Not the "double HistoryRun.complete" a quick read suggests: the success path and finalizeInterrupted are mutually exclusive (Phase 4 runs only if the wave block completed normally; the finalizer runs only if it was interrupted). The real defect is the opposite — a window where neither finalizes.

Root cause

src/main/scala/ccas/analysis/apps/history/HistoryApp.scala, discoverClub:

  • Phases 2–3 are wrapped: { ... } yield ws.copy(...) }.onInterrupt { finalizeInterrupted(...) } — the interrupt guard covers only this block.
  • Phase 4 (=== Finalize ===) runs after the block yields: reads the seed Refs, builds totalStats, calls HistoryRun.complete(runId, completedAt, ...), then logSummary(...), then yield HistoryResult(...).
  • The history_run row is inserted non-terminal in Phase 1 (HistoryRun.insert(...)), so until HistoryRun.complete runs the row is "started, not completed".

Interrupt timing within Phase 4:

  • Before HistoryRun.complete → row stays non-terminal forever; finalizeInterrupted does not fire (block already exited). A committed-but-incomplete run, indistinguishable from a crash.
  • Between HistoryRun.complete and logSummary → run marked complete but no summary logged (cosmetic; the per-job log just lacks its tail).

Why this is worse for History than Membership

Membership tolerates interrupts because reconcile's writes are one terminal transaction — an interrupt persists nothing, leaving only an incomplete membership_run that selectLatestCompleted ignores. History commits per-match work incrementally and completes the run as a separate step, so an interrupted Phase 4 leaves committed match data alongside a stranded non-terminal history_run.

Impact / severity

🟡 Low. The window is tiny (a few Ref.gets + one UPDATE + one log) and only opens on actual fiber interruption (server shutdown, JobRunner/scheduler cancel). Effect is mostly diagnostic: a stranded non-terminal history_run row and/or a missing log tail. No data loss — pending matches re-process next run. But the run-tracking table can accumulate phantom "incomplete" rows that aren't real crashes.

Proposed fix

Make Phase-4 finalization atomic w.r.t. interruption, and ideally unify it with finalizeInterrupted so there's one finalizer:

  • Minimal: wrap Phase 4 in ZIO.uninterruptible so once reached it runs to completion (HistoryRun.complete + logSummary).
  • Cleaner (structural): replace the .onInterrupt-on-the-block + separate-Phase-4 split with a single .onExit/ensuring finalizer over Phases 2–4 that completes the run from the Exit (success → full stats; interrupted → partial stats via the existing finalizeInterrupted body). Removes the duplicated HistoryRun.complete call site and closes the window by construction.

Either way: success-path and interrupt-path summaries stay mutually exclusive (current behaviour), and the run row always reaches a terminal state.

Refs: surfaced during review of the per-job-log reporting fix (follow-up to #130 / #133).

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:historyHistoryApp and match backfillbugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions