Noema required review can spend full budget on an invalid changed-line citation

[seonghobae]

Affected consumer

ContextualWisdomLab/naruon#1485 exact head 4e2cb3b39f89aff811d5113962861bd1963d2996.

Required Noema Review run 33444969216, job 99781918923, using trusted central workflow source 5686de41660d51a7a7f22b8840dfa6ccfe5ff3f1.

Exact evidence

The run had a hosted runner, verified the live exact PR head, minted the repository-scoped cwl-noema-review App token, booted the contextual-orchestrator sidecar, completed provider-route preflight, and entered scripts.ci.noema_review_gate.

After roughly 65 minutes in the model/review stage, the terminal deterministic failure was:

Noema reviewed line 1 is not an exact changed-side line

The product PR's repository-owned CI/security/dependency/Semgrep/Bandit/image workflows and coverage evidence independently completed successfully on the same exact head. This Noema result is non-passing, but it is not evidence that Naruon source line 1 is defective.

Root-cause class

The review pipeline allows a model-produced inline citation that does not map to an exact changed-side line to consume most of the required-review wall-clock before the deterministic publisher rejects it. The fail-closed rejection is correct; the latency and evidence classification are not useful enough for a required organization gate.

This is related to, but materially distinct from, #1611: #1611 covers a malformed adversarial-probe outcome followed by retry exhaustion/502. This issue covers changed-side line identity and inline-review publication validation.

Required repair

1. Add a deterministic RED fixture whose model review names a path/line that is not an exact changed-side line for the bound source head/live base.
2. Preserve fail-closed rejection; never coerce, nearest-line search, or silently relocate the finding to make it publishable.
3. Validate path/side/line identity as early as practical after each model result so an invalid citation does not consume the entire required-review budget.
4. Classify this as MODEL_OUTPUT_INVALID / review-unavailable evidence unless an independently valid source-backed finding exists; do not synthesize a source defect from the bad citation.
5. If retrying, send deterministic feedback describing the allowed changed-line contract and keep retries materially bounded. Preserve exact attempt receipts and final failure class.
6. Keep contextual-orchestrator as the only model-routing boundary; do not add direct-provider fallbacks.
7. Preserve exact-head/live-base binding, reviewer credential isolation, unresolved-thread behavior, and all deterministic security gates.
8. GREEN acceptance: rerun unchanged naruon#1485@4e2cb3b39f89aff811d5113962861bd1963d2996; the required Noema lane must either publish a schema-valid review whose inline findings all map to exact changed-side lines, or terminate within the bounded budget with typed model-output/review-unavailable evidence. Invalid citations must never be treated as source findings or passing review evidence.

[seonghobae]

Fresh second-repository reproduction on 2026-09-02 from ContextualWisdomLab/wardnet#141@fea3796a723080068cdc02e064065d6d53eeb3e0, base main@cc15cc2c34daf8c104eeb83d52a6a66f3cd6e128.

Required Noema Review run 33500964034, job 99833952859, used trusted central workflow source 960b08456de4c87a5a833938220d6d83f68d61c1. Runner acquisition, exact-head validation, repository-scoped reviewer App token, contextual-orchestrator sidecar provisioning, and provider preflight all succeeded. The sidecar eventually had 4 ready routes. The review stage then ran from about 17:58Z until 19:24Z and failed deterministically with exactly:

Noema reviewed line 1 is not an exact changed-side line

This Wardnet candidate changes one workflow action pin only. On the same exact head, repository-owned rust, dependency-review, Trivy, OSV, Semgrep, CodeQL and other deterministic checks executed successfully. There are no inline review threads and Devin reported no source issues. The Noema failure is therefore another central changed-line publication/validation failure, not evidence that Wardnet source line 1 is defective.

Please bind this exact repo/PR/head/base/run/job/trusted-workflow tuple to the #1613 RED fixture. GREEN should include this unchanged Wardnet head as a canary: either a schema-valid review whose inline citations are exact changed-side lines, or an early bounded typed MODEL_OUTPUT_INVALID / review-unavailable outcome. Do not nearest-line coerce, synthesize a source finding, or require leaf source churn.

[seonghobae]

Fresh Wardnet consumer evidence reproduces this same central defect on a different repository and newer trusted workflow source.

Affected exact candidate: ContextualWisdomLab/wardnet#147@b663f9d200e5f385c7dd067d074940a02836c68e, base main@cc15cc2c34daf8c104eeb83d52a6a66f3cd6e128.

Required Noema Review run 33550235981, job 99997877526, trusted central workflow source 696852eb44586150a4d9c22cba794b2089a9543a.

The job acquired ubuntu-24.04, verified the exact live PR head, minted the repository-scoped cwl-noema-review App token, materialized the trusted central source, booted contextual-orchestrator, and completed gateway/provider preflight. Wardnet's own exact-head CI 33550237523, Fuzz 33550237417, Security Scan 33550237618, and SAST Semgrep 33550237418 are independently terminal GREEN.

The first semantic-review failure was Noema reviewed line 2 is not an exact changed-side line. The bounded repair then exhausted its 900-second absolute deadline, terminating with NoemaRepairDeadlineExceeded. This is not a Wardnet source finding.

Please extend the existing RED/GREEN contract here to cover this exact combined class: valid-but-non-changed line citation -> bounded deterministic feedback -> repair deadline exhaustion. GREEN should reject/coerce nothing, classify the unavailable result distinctly from a source defect, terminate materially earlier than the full model budget when the changed-line contract remains invalid, preserve exact-head/live-base identity, and allow a fresh unchanged Wardnet #147 run to produce either a valid line-bound verdict or typed review-unavailable evidence without fabricating a finding.

[seonghobae]

Fresh Wardnet recurrence confirms the same causal class on ContextualWisdomLab/wardnet runner-pin candidate exact head b663f9d200e5f385c7dd067d074940a02836c68e (the same semantic runner-only delta now carried by PR #149; the failing required-review run was triggered while that head was on predecessor PR #147).

Exact required Noema run/job: 33550235981 / 99997877526, trusted central workflow source 696852eb44586150a4d9c22cba794b2089a9543a. Runner acquisition succeeded on ubuntu-24.04; exact-head validation, reviewer App token minting, contextual-orchestrator sidecar provisioning, provider-route preflight and gateway preflight all succeeded. The first causal review boundary was Prepare Noema model verdict, which ran from 20:33:19Z to 21:18:24Z and terminated with:

Noema bounded repair transport was exhausted; initial failure: Noema reviewed line 2 is not an exact changed-side line; repair failure: NoemaRepairDeadlineExceeded: Noema repair exceeded 900-second absolute wall-clock deadline

This is not a Wardnet source finding. Repository-owned exact-head Rust/CI, Fuzz, Semgrep and Security jobs have independently completed successfully on this unchanged SHA in earlier exact-head runs; the latest repeated runs are currently queue-starved and remain non-passing, tracked separately by #712.

Please include this Wardnet fixture in #1613 acceptance. RED should reproduce an invalid changed-side line such as line 2 against a tiny workflow-only diff. GREEN should reject that model output immediately or after a tightly bounded repair attempt, return typed MODEL_OUTPUT_INVALID/review-unavailable evidence without synthesizing a source defect, and avoid consuming ~45 minutes after the first deterministic line-identity failure. After central repair lands, Wardnet will rerun the unchanged exact head and require all central review findings to map to exact changed-side lines before merge classification.

[seonghobae]

Fresh Wardnet reproduction on an unchanged exact head strengthens this defect class.

Affected leaf: ContextualWisdomLab/wardnet#149@b663f9d200e5f385c7dd067d074940a02836c68e, protected base main@cc15cc2c34daf8c104eeb83d52a6a66f3cd6e128. Repository-owned CI/Fuzz/Security/SAST have already completed successfully on this exact source head, so this is not a Wardnet source/test failure.

Required Noema Review run 33550235981, job 99997877526, trusted central workflow SHA 696852eb44586150a4d9c22cba794b2089a9543a acquired an Ubuntu 24.04 hosted runner, verified the exact live head, minted the repository-scoped cwl-noema-review token, booted contextual-orchestrator and completed orchestrator/free preflight. It then terminated with:

Noema bounded repair transport was exhausted; initial failure: Noema reviewed line 2 is not an exact changed-side line; repair failure: NoemaRepairDeadlineExceeded: Noema repair exceeded 900-second absolute wall-clock deadline

This is the same causal class as the issue acceptance: an invalid model-produced changed-line identity correctly fails closed, but the repair path spends a full 900-second budget and the required gate ends as a generic failure instead of a bounded typed review-unavailable/model-output-invalid result.

RED: a model result cites line 2 when that line is not an exact changed-side line; deterministic validation rejects it, then repair exceeds the absolute deadline. GREEN: reject the invalid citation immediately, provide bounded contract feedback only within the declared repair budget, and terminate with typed MODEL_OUTPUT_INVALID/review-unavailable evidence unless a separately valid source-backed finding exists. Preserve exact-head/live-base binding and do not coerce the finding to a nearby line.

After central repair reaches protected truth, re-run this unchanged Wardnet exact head rather than mutating the leaf.

[seonghobae]

Fresh fast-mlsirm exact-head reproduction — same deterministic changed-line failure

A second current consumer reproduces this owner-lane defect after successful runner assignment and contextual-orchestrator preflight:

• target: ContextualWisdomLab/fast-mlsirm#1694
• exact unchanged source head: 8e6c30912685bc5d8991351ca4dea426d2386bdf
• protected target base at execution: main@45627700c26c29bca150896a9519a9b7426acb56
• Required Noema Review run/job: 33513290892 / 99873978260
• trusted central workflow source bound by that run: c70b081dd93cf9ca53c2277ba95eab0e200cbe5c
• hosted runner was assigned (ubuntu-24.04, eastus2), exact PR head validation succeeded, the repository-scoped reviewer App token was minted, and the contextual-orchestrator sidecar booted successfully.
• the sidecar reached orchestrator/free; gateway chat/completions preflight succeeded. Runtime evidence reported 12 probed routes, 3 ready routes, with the gateway itself status=ready.
• the first terminal causal boundary was the Noema publisher/validator after the model stage: Noema reviewed line 1 is not an exact changed-side line (process exit 1).

This is non-passing exact-head evidence, but it is not a fast-mlsirm source finding. The target source should not be moved or rewritten to accommodate a malformed model location. A rerun of the original workflow alone also must not be assumed to consume today's newer .github main: this failed run is immutably bound to trusted source c70b081….

RED/GREEN for this reproduction follows the issue contract: bind the model output to the exact current diff; reject invalid path/side/line coordinates without coercion; surface a typed MODEL_OUTPUT_INVALID/review-unavailable outcome when there is no independently valid finding; retry only with deterministic changed-line feedback and a bounded budget; and on a fresh unchanged-head dispatch, either publish a schema-valid authoritative Noema review whose inline locations are exact changed-side lines or fail closed with typed review-unavailable evidence. Do not fabricate relocation, drop a real finding, bypass the review gate, or transfer a predecessor verdict.

[seonghobae]

Fresh same-failure-class evidence from fast-mlsirm confirms the invalid changed-line path still consumes the bounded repair budget rather than terminating promptly with typed review-unavailable evidence.

Affected leaf: ContextualWisdomLab/fast-mlsirm#1697@00f5c7b1308f89aba5ffbe1663a4a715abde6a6a (protected base main@45627700c26c29bca150896a9519a9b7426acb56). Required Noema Review run 33397538166, attempt 3, job 99989735554, trusted central workflow source .github@0774e29acd7d4688fa2224f1c6fe6c56a03bbbb6.

The hosted runner and control-plane prerequisites all succeeded: exact live-head validation, repository-scoped cwl-noema-review token, target visibility, contextual-orchestrator sidecar, and gateway/provider preflight. The first causal failure is in Prepare Noema model verdict: Noema reviewed line 3 is not an exact changed-side line; the repair path then runs until NoemaRepairDeadlineExceeded: Noema repair exceeded 900-second absolute wall-clock deadline, after which publication is skipped and the required check fails. The sidecar itself had six ready free routes, so this is not a leaf checkout/build/scientific defect and not simple runner starvation.

RED discriminator: a model verdict cites a non-changed-side line on an exact-current PR and the gate enters repair long enough to exhaust the 900-second repair deadline. GREEN for this issue should include the unchanged fast-mlsirm exact head as an additional canary: invalid line identity is rejected immediately after each model result, any repair receives the exact allowed changed-line contract, retry count/wall time is materially bounded below the current failure horizon, and terminal failure is typed MODEL_OUTPUT_INVALID/review-unavailable rather than a leaf source finding. Preserve fail-closed semantics, exact-head binding, contextual-orchestrator-only routing, and no raw model-output logging.

[seonghobae]

Fresh downstream canary from ContextualWisdomLab/fast-mlsirm#1697, superseding leaf-side speculation for this failure class.

Exact consumer head is unchanged at 00f5c7b1308f89aba5ffbe1663a4a715abde6a6a; protected base observed at main@45627700c26c29bca150896a9519a9b7426acb56. Required Noema run 33397538166 attempt 3, job 99989735554, executed the trusted central workflow source ContextualWisdomLab/.github@0774e29acd7d4688fa2224f1c6fe6c56a03bbbb6.

All pre-model trust boundaries succeeded: hosted runner assignment, exact live-head validation, repository-scoped cwl-noema-review App token minting, public target-visibility resolution, contextual-orchestrator sidecar provisioning, and gateway preflight. The sidecar reported six ready routes despite bounded provider warnings, so this was not a preflight/credential/leaf-checkout failure. The first terminal causal boundary in Prepare Noema model verdict was:

Noema bounded repair transport was exhausted; initial failure: Noema reviewed line 3 is not an exact changed-side line; repair failure: NoemaRepairDeadlineExceeded: Noema repair exceeded 900-second absolute wall-clock deadline

The job then exited 1 without publishing a verdict. This is the same central changed-line validation class this issue owns, with an additional concrete proof that the bounded repair path can consume its full 900-second absolute deadline after the invalid citation has already been deterministically identified. It is not a fast-mlsirm source finding, and no leaf no-op/rewrite or gate weakening is appropriate.

RED acceptance from this canary: feed a schema-valid model verdict whose finding cites line 3 when line 3 is absent from the exact changed-side set, then exercise the repair transport under an unavailable/non-converging repair response. The gate must classify the initial evidence as model-output-invalid immediately and keep total repair work within the declared bounded budget without manufacturing a source defect.

GREEN acceptance: with central source repaired, rerun the unchanged then-current fast-mlsirm#1697 head. Noema must either publish a valid exact-head verdict whose line-bound findings all map to actual changed-side lines, or terminate promptly with typed review-unavailable/model-output-invalid evidence per the central fail-closed contract. Exact-head/live-base binding, reviewer credentials, contextual-orchestrator-only routing, and raw-output redaction must remain unchanged.

[seonghobae]

Fresh fast-mlsirm consumer evidence extends this same changed-line/repair-budget class to a newer trusted central workflow source, and separates the historical defect from the current protected fix.

Affected leaf: ContextualWisdomLab/fast-mlsirm#1479@4e9160b23b984b57a5fc2877ef02b4dbbef5d0f9, protected base main@b5a3a0c1057d4b53d7a4bb18e0de69f630c2b45c. Required Noema run 33640701505, job 100282742653, executed trusted .github@78271917b526469c559fa75cb5ee39426e5494d1 on an assigned Ubuntu 24.04 runner. Exact-head validation, the repository-scoped cwl-noema-review App token, target visibility, contextual-orchestrator sidecar provisioning, and gateway preflight all succeeded. The sidecar admitted 54 free routes and had 4 ready probes.

Prepare Noema model verdict then failed with the same causal sequence:

initial failure: Noema reviewed line entry 2/5 ... path='python/fast_mlsirm/factor_retention.py' line=145 side='RIGHT', which is not an exact changed-side line ... repair failure: NoemaRepairDeadlineExceeded: Noema repair exceeded 900-second absolute wall-clock deadline

Repository-owned CI, CodeQL, Semgrep, Security Scan, OSV and Scorecard are terminal GREEN on this unchanged leaf head, so this is not a factor-retention source/scientific failure.

Current protected .github/main@07d9ec23fb265c76539d23249e1dfa124ea7b23b has moved past that trusted workflow source: its doctoring and regression test explicitly treat the 900-second deadline as retired and forbid NOEMA_REPAIR_DEADLINE_SECONDS, _repair_wall_clock_deadline, NoemaRepairDeadlineExceeded, and signal.setitimer. I therefore dispatched a source-neutral rerun of the failed Noema job on the unchanged #1479 head. Do not count that as GREEN until the rerun demonstrates the current owner contract rather than replaying the old trusted workflow implementation.

GREEN remains: a fresh unchanged-head consumer canary must reject invalid changed-side identity without nearest-line coercion or invented source findings, route only through contextual-orchestrator, and either publish an exact-line-valid verdict or return the current typed review-unavailable/model-output-invalid contract without the retired fixed wall-clock repair deadline.