You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A pr-reviewer Stage 3 run was dispatched to a local OpenCode/MTPLX 27B model
with a 412 KB prompt (~100K tokens — the whole <cleared-public-review-input>
envelope is inlined by formatPublicReviewInputPrompt). The model's context
window is large enough (262144), so nothing rejected it, but prefill on local
hardware took roughly 9 minutes before the child emitted its first real line.
The run's own duration estimate was ~13 minutes. A prompt that spends most of the
budget in prefill either finishes with no margin or gets reaped mid-review, and
the whole stage is wasted — including the earlier stages' spend.
Observed on a live run: opencode run -m mtplx/<27b> at 0% CPU with the backend
at ~42%, no output for the first nine minutes, then normal tool activity.
Why nothing catches it today
server/services/prReviewerPipeline.js and server/lib/modelAbuseGuard.js
(formatPublicReviewInputPrompt) bound the material for the abuse scanner
(MODEL_ABUSE_GUARD_MAX_INPUT_CHARS = 2_000_000) but never for the reviewer's
prompt. Nothing compares the assembled prompt against the throughput of the
endpoint it is about to be dispatched to. server/lib/contextBudget.js already
exists and is used by the creative pipeline stages — the public-review path
does not use it.
What to build
Estimate the assembled Stage 3 prompt at dispatch time (estimateTokens
from server/lib/contextBudget.js).
When the target provider resolves to a local endpoint (providerBaseUrl
is already stamped onto the agent record for the per-endpoint spawn cap, Cap concurrent CoS agents per local inference endpoint #4834), compare the estimate against the run's duration budget using a
conservative local prefill rate, and either:
raise the run's duration estimate so the reaper does not kill a healthy
prefill, or
refuse the dispatch with an actionable blockedReason naming the prompt
size and the endpoint, so the stage falls back to a cloud provider rather
than burning the budget.
Whichever branch is chosen, surface the estimate on the agent record so the
card can show "large prompt — expect a long silent prefill" instead of an
apparently-idle run.
Decision already made, so this is ready to work: prefer (a) raising the
estimate — refusing outright would make a legitimately-configured local
provider unusable for this stage, and the user chose that provider deliberately.
Refusal is only right when the estimate exceeds the model's advertised context
window, which is a separate, already-fatal condition.
Not in scope
Trimming the envelope itself. The reviewer must see the complete screened
material; truncating it would produce a review of a partial patch.
Related
The UX half of this — the card gave no way to tell a long prefill from a wedged
run — is fixed separately: ANSI stripping on headless CLI output and a "No shell"
explanation chip on public-review agent cards.
Problem
A pr-reviewer Stage 3 run was dispatched to a local OpenCode/MTPLX 27B model
with a 412 KB prompt (~100K tokens — the whole
<cleared-public-review-input>envelope is inlined by
formatPublicReviewInputPrompt). The model's contextwindow is large enough (262144), so nothing rejected it, but prefill on local
hardware took roughly 9 minutes before the child emitted its first real line.
The run's own duration estimate was ~13 minutes. A prompt that spends most of the
budget in prefill either finishes with no margin or gets reaped mid-review, and
the whole stage is wasted — including the earlier stages' spend.
Observed on a live run:
opencode run -m mtplx/<27b>at 0% CPU with the backendat ~42%, no output for the first nine minutes, then normal tool activity.
Why nothing catches it today
server/services/prReviewerPipeline.jsandserver/lib/modelAbuseGuard.js(
formatPublicReviewInputPrompt) bound the material for the abuse scanner(
MODEL_ABUSE_GUARD_MAX_INPUT_CHARS = 2_000_000) but never for the reviewer'sprompt. Nothing compares the assembled prompt against the throughput of the
endpoint it is about to be dispatched to.
server/lib/contextBudget.jsalreadyexists and is used by the creative pipeline stages — the public-review path
does not use it.
What to build
estimateTokensfrom
server/lib/contextBudget.js).providerBaseUrlis already stamped onto the agent record for the per-endpoint spawn cap,
Cap concurrent CoS agents per local inference endpoint #4834), compare the estimate against the run's duration budget using a
conservative local prefill rate, and either:
prefill, or
blockedReasonnaming the promptsize and the endpoint, so the stage falls back to a cloud provider rather
than burning the budget.
card can show "large prompt — expect a long silent prefill" instead of an
apparently-idle run.
Decision already made, so this is ready to work: prefer (a) raising the
estimate — refusing outright would make a legitimately-configured local
provider unusable for this stage, and the user chose that provider deliberately.
Refusal is only right when the estimate exceeds the model's advertised context
window, which is a separate, already-fatal condition.
Not in scope
Trimming the envelope itself. The reviewer must see the complete screened
material; truncating it would produce a review of a partial patch.
Related
The UX half of this — the card gave no way to tell a long prefill from a wedged
run — is fixed separately: ANSI stripping on headless CLI output and a "No shell"
explanation chip on public-review agent cards.