feat(#677): add needs_input pushback for the code agent

[ralphbean]

Summary

• Add a needs_input field to the code-result schema so the agent can refuse to open a PR (broken sandbox tooling or a genuinely uninterpretable issue) and instead post an explanatory comment + fs-code-needs-input label.
• Extend eval/code/eval.yaml's pr_created judge to assert the negative when annotations.expect_pr: false, add a required_labels judge, and add eval case 002-push-back-on-nonsense covering the pushback path.

Test plan

• make check-bundle
• make test
• Watch CI (I didn't run the functional tests locally for this yet)

Closes #677

Assisted-by: Claude Opus 4.6 noreply@anthropic.com

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 8:11 PM UTC · Completed 8:28 PM UTC
Commit: 1ff174c · View workflow run →

[qodo-code-review]

PR Summary by Qodo

Add needs_input pushback path for the code agent (label + comment, no PR)

✨ Enhancement 🧪 Tests 📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

AI Description

• Add needs_input to the code agent result schema to support intentional “stop and ask” runs.
• Teach post-code to short-circuit: comment on the issue, apply fs-code-needs-input, and skip PR
 creation.
• Add schema + end-to-end post-script tests and an eval case asserting the no-PR + required label
 path.

Diagram

graph TD
  F(["Eval harness"]) --> A(["Code agent"]) --> B[/"agent-result.json"/] --> C(["post-code.sh" ભારે])
  C -->|"needs_input set"| D{{"GitHub Issue"}}
  C -->|"needs_input empty"| E{{"GitHub PR"}}
  subgraph Legend
    direction LR
    _p(["Process/script"]) ~~~ _f[/"JSON file"/] ~~~ _g{{"GitHub"}}
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Use a structured enum (needs_input_reason) + optional detail

• ➕ Allows better analytics/routing (e.g., tooling vs ambiguity)
• ➕ Enables automated remediation playbooks per reason
• ➖ Requires schema + script logic changes now and future reason taxonomy maintenance
• ➖ Still needs a freeform message field for actionable detail

2. Create a draft PR instead of refusing PR creation

• ➕ Keeps all context in a PR artifact reviewers are used to
• ➕ Allows attaching partial work or WIP commits
• ➖ Contradicts the goal of avoiding unverified/unsafe PRs (broken tooling, uninterpretable issues)
• ➖ Still needs label/comment signaling; increases noise in PR list

3. Use GitHub Checks / workflow annotations rather than issue labels

• ➕ More CI-native; can block merges and show status prominently
• ➕ Reduces label proliferation
• ➖ Harder to use as a human triage queue compared to labels
• ➖ More complex integration than the current post-script gh calls

Recommendation: The PR’s approach (single freeform needs_input string + deterministic post-script behavior: label, remove ready-to-code, comment, exit 0) is a strong default: it’s simple, human-actionable, and minimizes noise by avoiding PR creation. If future automation needs emerge, consider adding a needs_input_reason enum later while keeping the freeform message for specifics.

Files changed (15) +790 / -18

Enhancement (3) +118 / -0

code-result.schema.jsonAdd needs_input field to code agent result schema +6/-0

Add needs_input field to code agent result schema

• Introduces an optional 'needs_input' string with length bounds (1..4000) and documentation describing the no-commit, no-PR semantics when set.

schemas/code-result.schema.json

post-code.shImplement needs_input early exit (comment + label, no PR) in bundled script +56/-0

Implement needs_input early exit (comment + label, no PR) in bundled script

• Adds 'post_needs_input_comment' and an early-exit check after parsing agent output so runs with 'needs_input' stop cleanly before branch validation, git operations, or PR creation.

scripts/post-code.sh

post-code.src.shImplement needs_input early exit (source) for post-code script generation +56/-0

Implement needs_input early exit (source) for post-code script generation

• Adds the needs-input helper and short-circuit logic in the source script so regenerated bundles include the new behavior.

scripts/post-code.src.sh

Tests (2) +285 / -0

code-result-schema-test.shAdd schema validation tests for needs_input and regressions +96/-0

Add schema validation tests for needs_input and regressions

• Adds a focused test runner for 'schemas/code-result.schema.json' validating happy paths, unknown properties, missing required fields, and needs_input constraints.

scripts/code-result-schema-test.sh

post-code-needs-input-test.shAdd end-to-end test for post-code needs_input early exit +189/-0

Add end-to-end test for post-code needs_input early exit

• Runs the real 'post-code' script with a mocked 'gh' to assert it skips PR creation, applies the needs-input label, removes 'ready-to-code', posts a comment, exits 0, and respects 'CODE_NEEDS_INPUT_LABEL' overrides.

scripts/post-code-needs-input-test.sh

Documentation (4) +288 / -8

code.mdDocument needs_input structured output behavior for the code agent +4/-1

Document needs_input structured output behavior for the code agent

• Updates the structured output section to describe 'needs_input' as the mechanism to stop without committing and trigger an issue comment + label instead of a PR.

agents/code.md

code.mdAdd fs-code-needs-input control label documentation +1/-0

Add fs-code-needs-input control label documentation

• Documents the new 'fs-code-needs-input' label semantics, including when it is applied, that it removes 'ready-to-code', and how humans re-trigger after resolving the blocker.

docs/code.md

code-agent-needs-input.mdAdd design plan for needs_input pushback path +258/-0

Add design plan for needs_input pushback path

• Introduces a detailed design doc capturing problem statement, goals, schema/harness/script changes, and a TDD-oriented test plan for the needs_input behavior.

docs/plans/code-agent-needs-input.md

SKILL.mdUpdate code agent skill to use needs_input for blockers/ambiguity +25/-7

Update code agent skill to use needs_input for blockers/ambiguity

• Directs the agent to set 'needs_input' (and stop without committing) for missing scan-secrets, genuinely uninterpretable issues, and tooling/infra failures after one setup attempt—replacing the prior “commit with disclosure” guidance.

skills/code-implementation/SKILL.md

Other (6) +99 / -10

MakefileRun new post-code needs_input and schema test scripts +2/-0

Run new post-code needs_input and schema test scripts

• Adds the needs-input post-code test and a dedicated code-result schema test to the 'script-test' target so they run in the standard suite.

Makefile

annotations.yamlDefine eval annotations for needs_input (no PR expected) case +33/-0

Define eval annotations for needs_input (no PR expected) case

• Adds a new eval case annotation set that expects no PR and requires the 'fs-code-needs-input' label, with tighter budget targets for quick pushback.

eval/code/cases/002-push-back-on-nonsense/annotations.yaml

input.yamlAdd contradictory issue fixture to trigger needs_input pushback +21/-0

Add contradictory issue fixture to trigger needs_input pushback

• Creates an issue fixture with irreconcilable requirements to validate the agent refuses implementation and instead requests human clarification.

eval/code/cases/002-push-back-on-nonsense/input.yaml

repoPoint eval case at tiny-calc repo fixture +1/-0

Point eval case at tiny-calc repo fixture

• Adds the repo pointer file referencing the tiny-calc fixture used by the new eval case.

eval/code/cases/002-push-back-on-nonsense/repo

eval.yamlSupport no-PR eval cases and required label assertions +41/-10

Support no-PR eval cases and required label assertions

• Extends 'pr_created' judge to assert a negative when 'annotations.expect_pr: false', adds a 'required_labels' judge, and wires its threshold into the suite.

eval/code/eval.yaml

code.yamlConfigure CODE_NEEDS_INPUT_LABEL for the code harness +1/-0

Configure CODE_NEEDS_INPUT_LABEL for the code harness

• Adds 'CODE_NEEDS_INPUT_LABEL=fs-code-needs-input' to the runner environment so post-code can use a consistent label with an override fallback.

harness/code.yaml

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

1. Protected paths modified ✗ Dismissed 📜 Skill insight § Compliance

Description

This PR modifies protected governance/infrastructure paths (scripts/, skills/, harness/),
which must receive explicit human review and must not be auto-approved. Ensure appropriate
CODEOWNERS/maintainer approval before merging.

Code

scripts/post-code.src.sh[R80-83]

+post_needs_input_comment() {
+  local needs_input="$1"
+  local safe_issue_number
+  safe_issue_number="$(_sanitize_workflow_value "${ISSUE_NUMBER}")"

Relevance

●● Moderate

Protected-path governance is real, but prior “authorization note” style asks were rejected; unclear
what change is expected.

PR-#631

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Compliance requires raising a finding whenever protected governance/infrastructure paths are
modified. This PR adds/changes code under scripts/, modifies runner env configuration in
harness/, and updates agent skill instructions in skills/, so it must not be auto-approved and
needs human review.

scripts/post-code.src.sh[69-114]
harness/code.yaml[44-55]
skills/code-implementation/SKILL.md[42-47]
Skill: pr-review

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Protected governance/infrastructure paths were modified, which requires explicit human review and must not be auto-approved.

## Issue Context
The PR changes files under `scripts/`, `skills/`, and `harness/`, which are treated as protected paths.

## Fix Focus Areas
- scripts/post-code.src.sh[69-168]
- harness/code.yaml[44-55]
- skills/code-implementation/SKILL.md[42-47]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Needs-input PR judge loophole ✓ Resolved 🐞 Bug ≡ Correctness

Description

In eval/code/eval.yaml, pr_created treats expect_pr: false as “no OPEN/MERGED PR exists”, so a
regression that creates a PR and then closes it would still pass the needs_input case even though a
PR was opened.

Code

eval/code/eval.yaml[R153-156]

      prs = state.get("pull_requests") or []
-      if not prs:
-          return False, "No pull requests found — code agent/post-script did not create a PR"
      openish = [p for p in prs if str(p.get("state", "")).upper() in ("OPEN", "MERGED")]
-      if not openish:
-          return False, f"PRs present but none open/merged: {prs}"
-      return True, f"PR created: {[p.get('url') for p in openish]}"
+      expect_pr = outputs.get("annotations", {}).get("expect_pr", True)
+      if expect_pr:

Relevance

●●● Strong

Correctness gap in eval judge; tightening negative assertion matches repo’s tendency to harden eval
logic.

PR-#177

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The judge computes openish (OPEN/MERGED only) and, when expect_pr is false, it only fails if
openish is non-empty—ignoring CLOSED PRs entirely.

eval/code/eval.yaml[147-163]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The `pr_created` judge should enforce that *no PR was created at all* when `annotations.expect_pr: false`. The current logic only checks for absence of OPEN/MERGED PRs, which allows CLOSED PRs to slip through and weakens the regression guard for needs_input cases.

### Issue Context
This judge is used specifically to validate the needs_input pushback path, where the stated expectation is “No PR is opened”.

### Fix Focus Areas
- eval/code/eval.yaml[147-163]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Wrong needs-input label name ✓ Resolved 🐞 Bug ⚙ Maintainability

Description

agents/code.md claims the post-script applies a needs-input label, but the implemented and
documented control label is fs-code-needs-input, which can mislead humans and agents about the
actual workflow state/label to look for or remove.

Code

agents/code.md[R86-89]

+description, or `needs_input` when you need human input before you can
+proceed — in that case, do not commit, and the post-script applies a
+`needs-input` label and posts the text as an issue comment instead of
+opening a PR. The `code-implementation` skill describes the schema and

Relevance

●●● Strong

Teams usually accept doc clarifications to prevent agent/human misreads; aligns with prior agent-doc
fix patterns.

PR-#326

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The agent-facing docs say needs-input, but both the user docs and the post-script default to
fs-code-needs-input, so the name in agents/code.md is inconsistent with the actual behavior.

agents/code.md[84-90]
docs/code.md[34-39]
scripts/post-code.src.sh[80-94]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`agents/code.md` documents the wrong label name (`needs-input`) for the needs_input pushback path. The pipeline uses `fs-code-needs-input`, so this documentation mismatch can cause incorrect manual remediation steps and confusion.

### Issue Context
The correct label is documented in `docs/code.md` and is also the default used by the post-code script.

### Fix Focus Areas
- agents/code.md[84-90]
- (optional cross-check) docs/code.md[34-39]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

4. Silent label operation failures ✓ Resolved 🐞 Bug ◔ Observability

Description

post_needs_input_comment suppresses stderr and ignores failures for label create/apply/remove
operations, so the primary machine-readable signal (fs-code-needs-input) can silently fail while
the script still exits 0.

Code

scripts/post-code.src.sh[R88-92]

+  gh label create "${label}" --repo "${REPO_FULL_NAME}" \
+    --description "Code agent needs human input to proceed" --color "D93F0B" \
+    --force 2>/dev/null || true
+  gh api "repos/${REPO_FULL_NAME}/issues/${ISSUE_NUMBER}/labels" \
+    -f "labels[]=${label}" --silent 2>/dev/null || true

Relevance

●● Moderate

Repo sometimes prefers fail-closed, but this path is explicitly best-effort; no close precedent on
label ops.

PR-#415
PR-#38

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The needs_input handler discards errors for gh label create and gh api label add/remove, unlike
other scripts (e.g., triage) that fail or print errors when label application fails.

scripts/post-code.src.sh[80-114]
scripts/post-triage.sh[75-91]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
In the needs_input early-exit path, label create/apply/remove operations are all best-effort with `2>/dev/null || true` and no warnings. If permissions/transient GitHub errors occur, the run will look successful but the issue may not get the required label and may retain `ready-to-code`.

### Issue Context
Best-effort behavior is fine, but it should emit warnings (like the comment-posting failure path already does) so operators can diagnose why the label signal is missing.

### Fix Focus Areas
- scripts/post-code.src.sh[80-114]
- (contrast/reference) scripts/post-triage.sh[75-91]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context used

✅ Compliance rules (platform): 55 rules

✅ Skills: 4 invoked

  code-review

  code-implementation

  pr-review

  docs-review

_{To customize comments, go to the Qodo configuration screen, or learn more in the docs.}

[fullsend-ai-review]

Review

Findings

Medium

• [protected-path] agents/code.md, harness/code.yaml, scripts/code-result-schema-test.sh, scripts/post-code-needs-input-test.sh, scripts/post-code.sh, scripts/post-code.src.sh, skills/code-implementation/SKILL.md — This PR modifies 7 files under protected paths (agents/, harness/, scripts/, skills/). The PR links to issue Code agent needs a structured way to say 'needs human input' instead of silently no-oping #677 and the description explains the rationale for each change. Human approval is always required for protected-path changes, regardless of context.

Previous run

Review

Findings

High

• [logic error] skills/code-implementation/SKILL.md:896 — Step 11 (validate structured output) states "Only target_branch, pr_body, and closes_issue are allowed. Any other fields will cause validation to fail." This directly contradicts the new needs_input field added to the schema. When the agent writes needs_input and reaches step 11, this instruction tells it the field is disallowed, which could lead the agent to remove the field before validation — defeating the entire needs_input mechanism.
  Remediation: Update step 11 to list needs_input as an allowed optional property (e.g., "Only target_branch, pr_body, closes_issue, and needs_input are allowed.").

Medium

• [schema-compatibility] schemas/code-result.schema.json:25 — Adding an optional field to a schema with additionalProperties: false is backward-incompatible if downstream consumers (e.g., the fullsend CLI) validate against an older copy of the schema. Old validators will reject outputs containing needs_input even though the field is optional. This creates a deployment ordering constraint.
  Remediation: Verify the fullsend CLI fetches this schema at runtime (no pinned copy), or coordinate deployment order: update the CLI’s schema copy before merging this PR.

• [protected-path] agents/code.md, harness/code.yaml, scripts/post-code.sh, scripts/post-code.src.sh, scripts/code-result-schema-test.sh, scripts/post-code-needs-input-test.sh, skills/code-implementation/SKILL.md — This PR modifies 7 files under protected paths (agents/, harness/, scripts/, skills/). The PR links to issue Code agent needs a structured way to say 'needs human input' instead of silently no-oping #677 and the description explains the rationale for each change. Human approval is always required for protected-path changes, regardless of context.

Low

• [stale reference] skills/code-implementation/SKILL.md:637 — After replacing the "commit with disclosure" behavior with needs_input in step 9c, nearby text still says "If you cannot run the relevant test suite or lint command, you must disclose that." The phrasing assumes a commit-based disclosure, which is inconsistent with the new needs_input flow where the agent does NOT commit.
  Remediation: Update the text to reference needs_input as the expected action when the test/lint tool cannot run.

---

Labels: PR implements the needs_input pushback feature for the code agent, modifying agent definitions, harness config, post-scripts, skills, and eval infrastructure.

---

Next steps:

• /fs-fix — agent addresses review findings automatically
• /fs-fix <your instruction> — agent fixes with your specific guidance
• Push commits directly — review re-runs automatically on push
• /fs-fix-stop — disable automatic fix runs for this PR

[waynesun09]

Additional finding (no line in this PR's diff to anchor it to — schemas/code-result.schema.json line 7 isn't within the changed hunk):

[MEDIUM] target_branch kept unconditionally required, untested for the broken-tooling needs_input scenario — schemas/code-result.schema.json:7-8

The schema keeps required: ["target_branch"] unconditional even when needs_input is set. The PR's own design doc (docs/plans/code-agent-needs-input.md) justifies this only as "per current design the agent always writes target_branch regardless" and explicitly lists it under "Open items to watch during implementation" as an unconfirmed assumption, not a verified guarantee. The PR's stated motivation for needs_input is two-fold — (1) a genuinely uninterpretable issue and (2) broken sandbox tooling/environment — but only scenario (1) got an eval case (eval/code/cases/002-push-back-on-nonsense/); there is no case exercising a broken-environment run where the agent's normal means of determining target_branch (git/gh calls) might also fail. If that happens, agent-result.json fails schema validation, validation_loop skips post_script per ADR 0022, and the needs_input signal this feature exists to produce is lost silently — regressing to the pre-PR generic no-op.

Suggestion: Either add an eval case simulating broken tooling (unrelated to git/gh) to confirm target_branch is still reliably produced, or relax the schema so target_branch is optional when needs_input is set (e.g. via oneOf/if-then), since no push/PR happens on the needs_input path regardless of target_branch's value.

[ralphbean]

Re: #682 (comment)

Good catch on the schema-compliance line in SKILL.md step 11 — it still said only target_branch, pr_body, and closes_issue were allowed, which would've told the agent to strip needs_input right before validating. Fixed to list all four fields. Also updated the stale "you must disclose that" line near step 9c, which was left over from the old commit-with-disclosure behavior — it now points at needs_input instead.

On the protected-path note: intentional — this feature has to touch scripts/, harness/, and skills/ to exist at all.

The schema-compatibility point (optional field + additionalProperties: false being backward-incompatible for a stale CLI copy of the schema) is a real question but not one I can resolve unilaterally — flagging it for a human to confirm how the fullsend CLI resolves this schema at runtime.

[ralphbean]

Re: #682 (comment)

These four findings are the same ones raised inline — handled there: protected-path note dismissed as intentional, the needs-input → fs-code-needs-input label name fixed in agents/code.md, the pr_created judge tightened to reject any PR (not just open/merged) in the needs_input case, and warnings added to the label create/apply/remove calls.

[ralphbean]

Re: #682 (comment)

Following up on the schema-compatibility point — the schema ships bundled with this agent, not the CLI, so an old pinned CLI paired with the new agent would indeed reject needs_input. I think that's fine: if you're pinning the CLI and workflow versions, you should be pinning the agent version too.

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 10:09 PM UTC · Completed 10:23 PM UTC
Commit: 6d90896 · View workflow run →

Superseded by updated review

[waynesun09]

[MEDIUM] Contract-violation guard is comment-only, never fails the run

post_needs_input_comment() (added in 6d90896 as a fix for two earlier review threads) detects a genuine contract violation — the agent set needs_input but also left local unpushed commits or already has an open PR for the branch — and appends a warning sentence to the issue comment plus a gha_echo warning to the runner log. But the caller (scripts/post-code.sh:836-839) still exit 0s unconditionally right after calling this function, identical to the clean needs_input path. There's no distinct label, non-zero exit, or other machine-queryable signal — a maintainer/dashboard watching exit codes or labels can't distinguish "clean needs_input" from "agent violated the needs_input contract" without reading full comment prose.

Suggestion: keep posting the caveat prose, but also apply a separate marker (e.g. an fs-code-needs-input-conflict label) or route through the fail path so this state is alertable, not just buried text in a comment.

[waynesun09]

[MEDIUM] Silent default-branch fallback to "main" can mask the discarded-commits caveat it exists to produce

default_branch="$(gh api "repos/${REPO_FULL_NAME}" --jq '.default_branch' 2>/dev/null || echo main)" silently falls back to the literal string "main" on any API failure (auth hiccup, rate limit, transient network error). If the repo's actual default branch differs (e.g. "master") and the agent's branch genuinely has unpushed commits ahead of it, the following git rev-list --count "origin/${default_branch}..HEAD" either errors against a nonexistent ref (caught by || echo 0) or diffs against the wrong branch, so commits_ahead reports 0 and the "these commits were not pushed and will be discarded" caveat — the exact informative signal this block exists to surface — is silently dropped.

Suggestion: on API failure, log a warning that the discarded-commits check could not run instead of assuming "main", or derive the comparison branch more robustly (e.g. from the upstream tracking ref) rather than defaulting to a guessed branch name.

[waynesun09]

[MEDIUM] premature-decision: "one attempt only" setup-retry count for missing tooling is asserted without a cited requirement

The new guidance says: "try the Makefile's setup targets first (make deps, make setup, etc.) — one attempt only. If the tool still cannot run after that attempt, write needs_input...". Issue #677 and the PR description don't specify how many setup attempts the agent should make before giving up; this exact number appears to be a plausibility guess with no cited design-doc or incident basis. Too few attempts risks needs_input pushback (and the human-in-the-loop cost that entails) on transient/flaky setup failures a second attempt or alternate target would resolve; too many burns turn/cost budget on a genuinely broken sandbox.

Suggestion: cite where "one attempt" was decided, or soften to "a reasonable number of attempts (typically one, more only if the failure looks transient)" and let the eval harness gather real data before hard-coding a specific count.