Skip to content

Public-review Stage 3 dispatches a ~100K-token prompt to a local model without checking it fits the run's time budget #6117

Description

@atomantic

Problem

A pr-reviewer Stage 3 run was dispatched to a local OpenCode/MTPLX 27B model
with a 412 KB prompt (~100K tokens — the whole <cleared-public-review-input>
envelope is inlined by formatPublicReviewInputPrompt). The model's context
window is large enough (262144), so nothing rejected it, but prefill on local
hardware took roughly 9 minutes before the child emitted its first real line.

The run's own duration estimate was ~13 minutes. A prompt that spends most of the
budget in prefill either finishes with no margin or gets reaped mid-review, and
the whole stage is wasted — including the earlier stages' spend.

Observed on a live run: opencode run -m mtplx/<27b> at 0% CPU with the backend
at ~42%, no output for the first nine minutes, then normal tool activity.

Why nothing catches it today

server/services/prReviewerPipeline.js and server/lib/modelAbuseGuard.js
(formatPublicReviewInputPrompt) bound the material for the abuse scanner
(MODEL_ABUSE_GUARD_MAX_INPUT_CHARS = 2_000_000) but never for the reviewer's
prompt. Nothing compares the assembled prompt against the throughput of the
endpoint it is about to be dispatched to. server/lib/contextBudget.js already
exists and is used by the creative pipeline stages — the public-review path
does not use it.

What to build

  1. Estimate the assembled Stage 3 prompt at dispatch time (estimateTokens
    from server/lib/contextBudget.js).
  2. When the target provider resolves to a local endpoint (providerBaseUrl
    is already stamped onto the agent record for the per-endpoint spawn cap,
    Cap concurrent CoS agents per local inference endpoint #4834), compare the estimate against the run's duration budget using a
    conservative local prefill rate, and either:
    • raise the run's duration estimate so the reaper does not kill a healthy
      prefill, or
    • refuse the dispatch with an actionable blockedReason naming the prompt
      size and the endpoint, so the stage falls back to a cloud provider rather
      than burning the budget.
  3. Whichever branch is chosen, surface the estimate on the agent record so the
    card can show "large prompt — expect a long silent prefill" instead of an
    apparently-idle run.

Decision already made, so this is ready to work: prefer (a) raising the
estimate
— refusing outright would make a legitimately-configured local
provider unusable for this stage, and the user chose that provider deliberately.
Refusal is only right when the estimate exceeds the model's advertised context
window, which is a separate, already-fatal condition.

Not in scope

Trimming the envelope itself. The reviewer must see the complete screened
material; truncating it would produce a review of a partial patch.

Related

The UX half of this — the card gave no way to tell a long prefill from a wedged
run — is fixed separately: ANSI stripping on headless CLI output and a "No shell"
explanation chip on public-review agent cards.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

effort:mediumEffort: mediummodel:mediumModel size: mediumplanTracked by /do:replanplanner:opus-5Plan authored by the opus-5 model

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions