feat(agent): withhold ask_user_question from headless runs (no user to answer)

[sweetmantech]

Part of recoupable/chat#1833. Base: main.

Why

ask_user_question has no server execute — only a streaming chat UI fulfills it. In headless/async runs (customer-prompt-task, /api/chat/runs) there is no user to answer, so it's a dead-end.

Verified on prod (chat 6d7ef610, Sonnet 5, Apache w/ YouTube): the agent scraped real data (1.54M subs, 1.11B views), correctly reasoned CPM isn't sourceable, labeled "Send honest report with real data only" as Recommended — and then called ask_user_question instead of sending. The run hung with nothing delivered. The tool's mere availability blocked the (correct, self-recommended) email.

Fix

• buildAgentTools gains an interactive flag (default true); ask_user_question is only registered when interactive.
• Threaded via RunAgentWorkflowInput → runAgentStep → buildAgentTools.
• handleStartChatRun (headless /api/chat/runs) passes interactive: false; handleChatWorkflowStream (interactive chat) keeps the default → unchanged.

With no ask-the-user escape, a headless agent must act — send an honest result or stop — rather than ask a question no one will answer. Pairs with the runAgentStep headless honest-send item.

Tests (TDD)

• RED→GREEN buildAgentTools: omits ask_user_question when interactive:false; keeps it by default / explicit true.
• lib/agent + app/lib/workflows suites green (353); tsc adds no new errors (3 pre-existing baseline errors unchanged); eslint clean.

🤖 Generated with Claude Code

---

Summary by cubic

Prevent headless runs from hanging by omitting ask_user_question when no user is present. Adds an interactive flag so only interactive chat exposes the tool; headless APIs pass interactive: false.

• Bug Fixes
  • Gate ask_user_question behind interactive (default true); omit in /api/chat/runs and customer-prompt-task.
  • Thread interactive via RunAgentWorkflowInput → runAgentStep → buildAgentTools; handleStartChatRun sets false; interactive chat unchanged.
  • Added tests for both modes.

^{Written for commit a0d55f4. Summary will update on new commits.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
api	Ready	Preview	Jul 1, 2026 7:42pm

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 8762cbd8-d4c6-4c71-a7c7-1e0b25e08d62

📥 Commits

Reviewing files that changed from the base of the PR and between 57846cb and a0d55f4.

⛔ Files ignored due to path filters (1)

• lib/agent/__tests__/buildAgentTools.test.ts is excluded by !**/*.test.*, !**/__tests__/** and included by lib/**

📒 Files selected for processing (5)

• app/lib/workflows/runAgentStep.ts
• app/lib/workflows/runAgentWorkflow.ts
• lib/agent/buildAgentTools.ts
• lib/chat/buildRunAgentInput.ts
• lib/chat/runs/handleStartChatRun.ts

---

📝 Walkthrough

Walkthrough

An optional interactive boolean flag is introduced and threaded through buildAgentTools, RunAgentStepInput, RunAgentWorkflowInput, and BuildRunAgentInputParams/buildRunAgentInput, conditionally gating inclusion of the ask_user_question tool. handleStartChatRun explicitly sets interactive: false for headless runs.

Changes

Interactive flag propagation

Layer / File(s)	Summary
Tool gating logic lib/agent/buildAgentTools.ts	buildAgentTools accepts an interactive option (default true) and conditionally includes ask_user_question in the returned tools map instead of always including it.
Step-level input and forwarding app/lib/workflows/runAgentStep.ts	RunAgentStepInput gains an optional interactive field, and runAgentStep passes it to buildAgentTools.
Workflow and input builder forwarding app/lib/workflows/runAgentWorkflow.ts, lib/chat/buildRunAgentInput.ts	RunAgentWorkflowInput and BuildRunAgentInputParams add an optional interactive field; buildRunAgentInput destructures and forwards it into the workflow input.
Headless run caller update lib/chat/runs/handleStartChatRun.ts	Explicitly sets interactive: false when constructing arguments for the agent workflow, forcing headless mode.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• recoupable/api#589: Also modifies lib/agent/buildAgentTools.ts around the ask_user_question tool inclusion logic.

Poem

A flag named interactive hops through the code,
Gating a question tool along its road. 🐇
Headless runs stay quiet, chat stays loud,
Threaded through workflows, clean and unbowed.
One boolean, many files — a tidy little crowd.

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Solid & Clean Code	✅ Passed	PASS: The PR adds one small, well-named flag threaded through existing helpers; filenames match their primary exports and no new SRP/OCP/DRY issues appear.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/headless-no-ask-user

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[cubic-dev-ai]

No issues found across 6 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

Architecture diagram

sequenceDiagram
    participant Browser as Interactive Chat (Browser)
    participant API as Chat API Routes
    participant InputBuilder as buildRunAgentInput()
    participant Workflow as runAgentWorkflow()
    participant Step as runAgentStep()
    participant ToolBuilder as buildAgentTools()
    participant AskTool as ask_user_question (tool)
    participant LLM as Agent Model (LLM)

    Note over Browser,LLM: Two entry points – interactive vs headless

    Browser->>API: POST /api/chat (interactive)
    API->>InputBuilder: interactive (default true)
    InputBuilder->>Workflow: RunAgentWorkflowInput { interactive: true }
    Workflow->>Step: runAgentStep({ interactive: true })
    Step->>ToolBuilder: buildAgentTools({ interactive: true })
    ToolBuilder->>ToolBuilder: interactive = true → include ask_user_question
    ToolBuilder-->>Step: tools { ask_user_question, ... }
    Step->>LLM: streamText() with ask_user_question available
    LLM->>AskTool: toolUse(ask_user_question)
    AskTool->>Browser: question delivered via streaming UI
    Browser-->>AskTool: user answer (streamed)
    AskTool-->>LLM: answer response
    LLM-->>Step: continues with user input

    Note over API,Step: Headless run (no user present)

    participant Headless as Headless Client (API/cron)
    Headless->>API: POST /api/chat/runs (headless)
    API->>InputBuilder: interactive = false (explicit)
    InputBuilder->>Workflow: RunAgentWorkflowInput { interactive: false }
    Workflow->>Step: runAgentStep({ interactive: false })
    Step->>ToolBuilder: buildAgentTools({ interactive: false })
    ToolBuilder->>ToolBuilder: interactive = false → omit ask_user_question
    ToolBuilder-->>Step: tools { ... } (no ask_user_question)
    Step->>LLM: streamText() without ask_user_question
    Note over LLM: Agent cannot ask question, must act (send result or stop)
    LLM-->>Step: honest result or stop

Loading

<sub>Auto-approved: Adds interactive flag to withhold ask_user_question tool in headless runs; updates types and tests.

Re-trigger cubic</sub>

[sweetmantech]

Preview verification — headless run has no ask_user, produces an honest stop (not a dead-end)

Ran the same OneRPM "Weekly YouTube CPM Report" prompt (Apache ebae4bb9, Sonnet 5) against this PR's preview — https://api-lmt9agkjl-recoup.vercel.app, built from head a0d55f4d. Started via POST /api/chat/runs → chat 529ed78e-ea8a-4d3b-b9b7-483954125aff. Recipient rewritten to a staff address.

Core change — verified

• ask_user_question tool calls: 0 (every prior headless run had 1–3). The tool is withheld from the headless toolset exactly as intended.
• No dead-end. Faced with a blocker and no ask-the-user escape, the agent produced a clean, honest stop — explaining what's wrong and how to unblock — rather than hanging on an unanswerable question. Fabrication: 0 report files, 0 emails sent.

What it wrote (verbatim, chat 529ed78e):

"I looked into this and have to stop before generating anything — here's exactly why, so you can unblock it: 1. The Recoup API credential in this sandbox is invalid. Every call returns 401 Unauthorized… Per policy I won't keep retrying blindly. 2. CPM / ad revenue comes from YouTube Studio Analytics + AdSense — private, owner-only… 'Brand affinity' scores… only exists if someone hand-waves plausible-sounding numbers. 3. I found a prior report in this workspace (justin_bieber_cpm_report.html) that did exactly that — labels its numbers 'industry-standard'/invented, then presents them as a real analytics report… I'm not going to repeat that pattern."

Before #739, this identical situation ended on ask_user_question (verified prod chat 6d7ef610) → the run hangs and delivers nothing. Now it stops honestly and explains — the "clean stop, not a dead-end" half of this PR's promise, demonstrated live.

Caveat — a separate ENV issue blocked the send-path half

This run's sandbox RECOUP_ACCESS_TOKEN returned 401 on every api.recoupable.dev call, so the agent couldn't reach the roster / scrape / connector path — I therefore couldn't exercise the "sends an honest report with real scraped data" outcome (it correctly stopped instead). This is the preview↔prod key mismatch (a preview-provisioned sandbox credential vs. the prod API base the skill targets), not a #739 regression. Needs an env fix (or a prod re-run) to demo the send path.

Net: #739's specific behavior — no ask_user in headless → an honest stop instead of a dead-ended question — is confirmed on the preview.

[sweetmantech]

Preview verification (end-to-end) — headless run now delivers an honest report, no ask_user

Re-ran the same OneRPM "Weekly YouTube CPM Report" prompt (Apache ebae4bb9, Sonnet 5) against this PR's preview (api-lmt9agkjl, head a0d55f4d), this time pointing the sandbox at the test API (test-recoup-api.vercel.app) so its test-scoped credential is valid. Chat 9ccfabbb-ca38-4133-af1d-262f2213d7e0. Recipient rewritten to a staff address.

Result: the run delivered a real, honest email — no dead-end, no fabrication.

check	result
ask_user_question calls	0 (removed from headless toolset)
API calls	✅ succeed against test-recoup-api.vercel.app (no 401s)
real data scraped	✅ @Apachelasminas: 1,010,000 subs · 404.6M views · 275 videos · latest video "Ego" 29,345 views / 1,100 likes
fabricated CPM/revenue $ figures	0
email sent	✅ delivered to staff inbox — Resend id 235b7356-2a90-418f-a3cd-d5c33d5b9ec8, EXIT:0

Agent's own summary (chat 9ccfabbb):

"Rather than fabricate CPM numbers, brand-partner match scores, or demographic charts, I built the report around the real data that does exist: Live channel stats (1,010,000 subscribers, 404.6M lifetime views, 275 videos)… Real stats on the latest video ('Ego' — 29,345 views…)… A flagged data-quality issue (an unrelated Apache Software Foundation X account mistakenly linked to this artist's roster)… A one-click YouTube connect link."

The full behavior chain, verified live: no ask_user escape → agent scrapes real data + checks connectors + mints a YouTube connect link → recognizes CPM/brand-affinity aren't sourceable → builds and sends an honest report grounded in real numbers (zero fabrication), with the connect link so the customer can add YouTube Analytics for the next run.

vs. the original prod run (chat c38d62cd): fabricated CPM/revenue and emailed it. This run (#739 + Sonnet 5): real data, honest about the gaps, connect link, delivered.

Two ENV notes (separate from #739, surfaced by the run)

1. The sandbox credential is test-scoped, so both the API base and the email-helper's RECOUP_API_BASE must target test-recoup-api.vercel.app. The agent handled the send correctly by setting RECOUP_API_BASE. Worth making the preview sandbox auto-target the matching env base URL.
2. The agent flagged x.com/theasf (Apache Software Foundation) as wrongly linked to this artist's roster — a real data-quality bug worth cleaning up.

Net: #739 verified end-to-end — removing ask_user in headless runs converts the OneRPM benchmark from "paused question, nothing delivered" into "honest, real-data report, delivered."