fix(orchestrator): record why an agent reported a task 'not needed' - #1195
Merged
Conversation
`SkipReason.AgentNotNeeded` records that the agent decided the skip, not what it decided, and `complete_task` offers `not needed` both for "the step does not apply to this project" and for "you cannot do it". For the one step that stops to ask the user for credentials those are opposite outcomes, and both land in a bucket documented as the first. Add an optional `notNeededReason` to `complete_task` on both harness schemas — a closed set of `not-applicable` / `user-declined` / `blocked` — carry it on the task, and stamp it onto `orchestrator task skipped` as an additive property. `reason` and every other property keep their current meaning. The value is validated against the declared set before it reaches the task: the pi harness passes tool arguments through unvalidated, and free text about this step can name a credential. Generated-By: PostHog Desktop Task-Id: 93b396ed-9a30-4d56-bd97-6bd31c232436
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Test against a Context Mill branch:
Add Results will be posted here when complete. |
edwinyjlim
approved these changes
Sep 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
orchestrator task skippedcarries areason, andagent-not-neededisdocumented in
queue.tsas "the step did not apply here". But that valuerecords who decided the skip, not what they decided — and
complete_task's own guidance offersnot neededfor two different things:For most steps that conflation is harmless. For the seeded data-source step it
is not, because that is the one step that stops to ask the user for
credentials. An agent that asked, got nothing back, and handed the user a
browser link reports
not needed— and is then counted as having found thatthe step did not apply to a project the wizard had already scanned and found
connectable sources in. By construction "did not apply" is almost never true
for this step, yet it is the bucket most of its skips land in.
The result is the majority of this step's skips sitting in a bucket whose
recorded meaning is wrong for them, and a funnel that cannot separate "nothing
here to connect" from "the user withheld a credential" from "an endpoint we
could not reach". Those want three different product responses.
Why
Raised from a read of this step's telemetry: agent-reported skips dominate its
skip count, while the agents' own end-of-task remarks describe cancelled
credential prompts and browser fallbacks — not inapplicable steps. The event
had no field that could tell the two apart.
Changes
NotNeededReasoninqueue.ts: a closed set ofnot-applicable/user-declined/blocked, recorded on the task next toskipReason.complete_taskgains an optionalnotNeededReasonon both harnessschemas — the zod shape and the pi typebox mirror. The orchestrator runs on
pi, so a field present only on the MCP schema would not be in effect; the
complete_taskargs now share one shape per harness and the parity testholds them level, the same way it already does for the handoff.
orchestrator task skippedgainsnot_needed_reason. Purely additive:reasonand every other property are untouched, so existing dashboards keepreading the same numbers.
The pi harness hands tool arguments over unvalidated and an agent asked for
a reason readily writes a sentence — a sentence about this step can name a
database or a key, so it must not become an analytics dimension. Handoff
prose still stays out of telemetry entirely, exactly as before.
Deliberately unchanged: which status an agent picks. Rewording the status
guidance would move runs between
done/not needed/failedand breakthe comparison this event is being read for. This adds the missing dimension
without moving anything.
Test plan
pnpm build && pnpm test && pnpm lint— 2599 tests pass, 0 lint errors (theone warning in
queue-tools.tspredates this branch, verified againstmain).New tests:
queue.test.ts— each reason is recorded alongside the skip reason, staysunset when none is declared, reaches the transition listener (so the event
carries it), and the type guard rejects free text
queue-tools.test.ts—applyCompleteforwards a declared reason, drops anundeclared value, skips as before with none, the ask names every reason, and
both harnesses offer the field
Created with PostHog Desktop