Skip to content

orb(selfhost): deploy restarts kill passes between surface-publish and disposition; repair sweep can't see the orphan #8997

Description

@JSONbored

Problem

A selfhost deploy restarts the ORB container with no drain: SIGTERM kills whatever PR pass is in flight at an arbitrary point. When the kill lands between pr_public_surface_published and the disposition plan/execute, the PR is left wearing a decisive panel ("blocked / fixes required", red CI) while the matching disposition (the close) never executes — exactly the "ORB posted a closure review but the PR is still open" state.

Verified live 2026-07-26 during the beta.6 rollout (container replaced at 14:55:30Z):

The boot-time outage-repair sweep keys on "PR missing a current-head surface" (surfaceRepairPriorityPullNumbers), so a PR with a published current-head surface but no disposition for that head looks healthy to it — the 14:58:14Z regate sweep reported "recomputed 12 stale open PR verdict(s); 0 flagged" and skipped #8965. Recovery only happened via ordinary re-gate staleness minutes later.

Every deploy that lands during active contributor hours reproduces a window of these. With deploys happening several times a day right now, maintainers keep seeing "decisive panel, no action".

Fix

Two complementary parts:

  1. Graceful shutdown: on SIGTERM, stop claiming new jobs and let the in-flight PR pass finish (bounded, e.g. 30–60s grace), or abort it before the publish step — publish + disposition should behave as one unit. Killing after publish but before disposition is the worst possible cut point and is currently the likeliest one (publish happens early, the disposition tail is slow).
  2. Boot-time disposition reconciliation: at startup (and/or in the repair sweep), flag any open PR whose current-head surface exists but has no disposition audit event for that head and enqueue an immediate re-gate for it. The re-run is cheap — the review comes from one-shot reuse; only the disposition tail executes.

Acceptance

  • Kill -TERM during a pass between publish and disposition → after restart, the PR is re-disposed within one sweep tick (≤ ~2 min), not ~12 min.
  • Repair sweep audit explicitly reports "surface-without-disposition" repairs as their own reason.

Related: #8965 (evidence), the orphaned ai-review-lock sibling issue (filed alongside this one).

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.orbGittensory Orb related - maintainer self-hosting analytics.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions