feat: Add explain and plan commands (Seer AI) #39
Conversation
Codecov Results 📊❌ Patch coverage is 78.60%. Project has 1775 uncovered lines. Files with missing lines (22)
Coverage diff@@ Coverage Diff @@
## main #PR +/-##
==========================================
- Coverage 63.80% 62.56% -1.24%
==========================================
Files 35 40 +5
Lines 3956 4741 +785
Branches 0 0 —
==========================================
+ Hits 2524 2966 +442
- Misses 1432 1775 +343
- Partials 0 0 —Generated by Codecov Action |
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
Adds Seer (AI) integration to the CLI by introducing Autofix + Issue Summary types, API client methods, formatters, and new sentry issue explain / sentry issue fix commands, with accompanying Bun tests.
Changes:
- Add Zod schemas + helpers for the Seer Autofix API and Issue Summary API responses.
- Add CLI commands/utilities and formatters for showing root cause analysis, progress, and summaries.
- Add Bun test coverage for new API client methods, formatters, and polling logic.
Reviewed changes
Copilot reviewed 16 out of 17 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| test/types/autofix.test.ts | Adds unit tests for Autofix type helpers (terminal status, progress/root cause/PR URL extraction). |
| test/lib/formatters/summary.test.ts | Adds tests for formatting Issue Summary output. |
| test/lib/formatters/autofix.test.ts | Adds tests for Autofix output formatting, progress messaging, and error mapping. |
| test/lib/api-client.autofix.test.ts | Adds mocked-fetch tests for Autofix + Issue Summary API client methods. |
| test/commands/issue/utils.test.ts | Adds tests for issue ID/org resolution and Autofix polling behavior. |
| src/types/sentry.ts | Introduces Zod schemas/types for Issue Summary responses. |
| src/types/index.ts | Re-exports new Autofix and Issue Summary types/schemas via the public types barrel. |
| src/types/autofix.ts | Adds Autofix Zod schemas/types plus helper extractors (root causes, progress, PR URL, solution). |
| src/lib/formatters/summary.ts | Adds human-readable formatting for Issue Summary output. |
| src/lib/formatters/index.ts | Re-exports new Autofix/Summary formatter modules. |
| src/lib/formatters/autofix.ts | Adds Autofix-specific progress, root cause, status, error, and solution formatting. |
| src/lib/api-client.ts | Adds API client methods for triggering/polling/updating Autofix and fetching Issue Summary. |
| src/commands/issue/utils.ts | Adds shared issue utilities: org+issue resolution and Autofix polling with spinner. |
| src/commands/issue/index.ts | Registers new issue explain and issue fix subcommands and updates command docs. |
| src/commands/issue/fix.ts | Implements sentry issue fix command logic (validate analysis, choose cause, continue run, render solution). |
| src/commands/issue/explain.ts | Implements sentry issue explain command logic (trigger/poll analysis, render causes). |
| bun.lock | Updates dependencies lockfile (notably @types/node version). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
src/commands/issue/fix.ts
Outdated
| // Update autofix to continue to PR creation | ||
| await updateAutofix(org, numericId, state.run_id); |
There was a problem hiding this comment.
The selected root cause (causeId / selectedCause) is validated and shown to the user, but it is never sent to the API. As a result, --cause has no effect and the backend can’t know which root cause to proceed with. Update the autofix update call to include the chosen cause (e.g., send cause_id / selection payload alongside run_id) and thread that through updateAutofix’s API client signature.
| // Update autofix to continue to PR creation | |
| await updateAutofix(org, numericId, state.run_id); | |
| // Update autofix to continue to PR creation, including the selected cause | |
| await updateAutofix(org, numericId, state.run_id, causeId); |
| export async function getAutofixState( | ||
| orgSlug: string, | ||
| issueId: string | ||
| ): Promise<AutofixState | null> { | ||
| const response = await apiRequest<AutofixResponse>( | ||
| `/organizations/${orgSlug}/issues/${issueId}/autofix/` | ||
| ); | ||
|
|
||
| return response.autofix; |
There was a problem hiding this comment.
getAutofixState doesn’t validate the response payload. Since you already have AutofixResponseSchema, pass it as schema to apiRequest so malformed/changed API responses are caught early instead of propagating unknown shapes through the CLI.
src/lib/api-client.ts
Outdated
| * @param payload - The update payload (select_root_cause, select_solution, create_pr) | ||
| * @returns The response from the API | ||
| */ | ||
| export function updateAutofix( | ||
| orgSlug: string, | ||
| issueId: string, | ||
| runId: number | ||
| ): Promise<unknown> { | ||
| return apiRequest(`/organizations/${orgSlug}/issues/${issueId}/autofix/`, { | ||
| method: "POST", | ||
| body: { | ||
| run_id: runId, | ||
| step: "solution", | ||
| }, |
There was a problem hiding this comment.
The JSDoc for updateAutofix documents a payload parameter, but the function signature and implementation don’t accept or use it. Either add the payload parameter and forward it to the API request body, or update the docs to match the actual API contract.
| * @param payload - The update payload (select_root_cause, select_solution, create_pr) | |
| * @returns The response from the API | |
| */ | |
| export function updateAutofix( | |
| orgSlug: string, | |
| issueId: string, | |
| runId: number | |
| ): Promise<unknown> { | |
| return apiRequest(`/organizations/${orgSlug}/issues/${issueId}/autofix/`, { | |
| method: "POST", | |
| body: { | |
| run_id: runId, | |
| step: "solution", | |
| }, | |
| * @param payload - The update payload (select_root_cause, select_solution, create_pr). If omitted, defaults to { step: "solution" }. | |
| * @returns The response from the API | |
| */ | |
| export function updateAutofix( | |
| orgSlug: string, | |
| issueId: string, | |
| runId: number, | |
| payload?: Record<string, unknown> | |
| ): Promise<unknown> { | |
| const body = payload | |
| ? { | |
| run_id: runId, | |
| ...payload, | |
| } | |
| : { | |
| run_id: runId, | |
| step: "solution", | |
| }; | |
| return apiRequest(`/organizations/${orgSlug}/issues/${issueId}/autofix/`, { | |
| method: "POST", | |
| body, |
src/lib/api-client.ts
Outdated
| export function updateAutofix( | ||
| orgSlug: string, | ||
| issueId: string, | ||
| runId: number | ||
| ): Promise<unknown> { | ||
| return apiRequest(`/organizations/${orgSlug}/issues/${issueId}/autofix/`, { | ||
| method: "POST", | ||
| body: { | ||
| run_id: runId, | ||
| step: "solution", | ||
| }, | ||
| }); |
There was a problem hiding this comment.
updateAutofix currently hardcodes { step: "solution" } and returns Promise<unknown>, which makes it easy to accidentally ship an incorrect request shape (and it can’t support selecting a root cause or creating a PR). Consider accepting a typed AutofixUpdatePayload (and/or explicit step/stopping point) and validating the response with a Zod schema, similar to triggerAutofix.
| export function updateAutofix( | |
| orgSlug: string, | |
| issueId: string, | |
| runId: number | |
| ): Promise<unknown> { | |
| return apiRequest(`/organizations/${orgSlug}/issues/${issueId}/autofix/`, { | |
| method: "POST", | |
| body: { | |
| run_id: runId, | |
| step: "solution", | |
| }, | |
| }); | |
| const AutofixUpdateRequestSchema = z.object({ | |
| run_id: z.number(), | |
| step: z.enum(["select_root_cause", "select_solution", "create_pr"]).optional(), | |
| root_cause_id: z.string().optional(), | |
| solution_id: z.string().optional(), | |
| }); | |
| type AutofixUpdateRequest = z.infer<typeof AutofixUpdateRequestSchema>; | |
| export type AutofixUpdatePayload = Omit<AutofixUpdateRequest, "run_id">; | |
| export function updateAutofix( | |
| orgSlug: string, | |
| issueId: string, | |
| runId: number, | |
| payload: AutofixUpdatePayload = { step: "select_solution" } | |
| ): Promise<AutofixTriggerResponse> { | |
| const requestBody = AutofixUpdateRequestSchema.parse({ | |
| run_id: runId, | |
| ...payload, | |
| }); | |
| return apiRequest<AutofixTriggerResponse>( | |
| `/organizations/${orgSlug}/issues/${issueId}/autofix/`, | |
| { | |
| method: "POST", | |
| body: requestBody, | |
| schema: AutofixTriggerResponseSchema, | |
| } | |
| ); |
src/commands/issue/utils.ts
Outdated
| let currentMessage = "Waiting for analysis to start..."; | ||
|
|
||
| // Animation timer runs independently of polling for smooth spinner | ||
| let animationTimer: Timer | undefined; |
There was a problem hiding this comment.
Timer isn’t declared in this module and can be environment/type-package dependent. Prefer ReturnType<typeof setInterval> (or NodeJS.Timeout if you want Node-specific) so the interval handle type stays correct across Bun/Node without relying on a global Timer name.
| let animationTimer: Timer | undefined; | |
| let animationTimer: ReturnType<typeof setInterval> | undefined; |
src/lib/formatters/summary.ts
Outdated
| if (summary.scores?.possibleCauseConfidence !== null) { | ||
| const confidence = summary.scores?.possibleCauseConfidence; | ||
| if (confidence !== undefined) { | ||
| const percent = Math.round(confidence * 100); | ||
| lines.push(muted(`Confidence: ${percent}%`)); | ||
| } |
There was a problem hiding this comment.
The confidence check is harder to read than necessary and relies on a !== null guard plus a second undefined check. Consider simplifying to a single typeof confidence === "number" (or confidence != null) to make the intent clearer and avoid treating undefined as a meaningful value.
| if (summary.scores?.possibleCauseConfidence !== null) { | |
| const confidence = summary.scores?.possibleCauseConfidence; | |
| if (confidence !== undefined) { | |
| const percent = Math.round(confidence * 100); | |
| lines.push(muted(`Confidence: ${percent}%`)); | |
| } | |
| const confidence = summary.scores?.possibleCauseConfidence; | |
| if (typeof confidence === "number") { | |
| const percent = Math.round(confidence * 100); | |
| lines.push(muted(`Confidence: ${percent}%`)); |
src/types/autofix.ts
Outdated
| if (!state.steps) { | ||
| return; | ||
| } | ||
|
|
||
| // Look for PR info in steps or coding_agents | ||
| for (const step of state.steps) { |
There was a problem hiding this comment.
extractPrUrl returns early when state.steps is undefined, but PR information can still exist under state.coding_agents (as your own logic later supports). This early return prevents extracting the PR URL in that case. Consider removing the early return and instead treating missing steps as an empty list so coding_agents can still be checked.
| if (!state.steps) { | |
| return; | |
| } | |
| // Look for PR info in steps or coding_agents | |
| for (const step of state.steps) { | |
| // Look for PR info in steps or coding_agents | |
| const steps = state.steps ?? []; | |
| for (const step of steps) { |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 14 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| } | ||
|
|
||
| // Trigger solution planning to continue to PR creation | ||
| await triggerSolutionPlanning(org, numericId, state.run_id); |
There was a problem hiding this comment.
--cause/causeId is validated and displayed, but it is never sent to the API. As a result, selecting a root cause has no effect on what the backend plans. Add an API call/payload to select the root cause (and any stopping point) before triggering solution planning, or include the selected cause in the request if the endpoint supports it.
| await triggerSolutionPlanning(org, numericId, state.run_id); | |
| await triggerSolutionPlanning(org, numericId, state.run_id, { causeId }); |
src/commands/issue/plan.ts
Outdated
| brief: "Create a PR with a plan using Seer AI", | ||
| fullDescription: | ||
| "Create a pull request with a plan for a Sentry issue using Seer AI.\n\n" + | ||
| "This command requires that 'sentry issue explain' has been run first " + | ||
| "to identify the root cause. It will then generate code changes and " + | ||
| "create a pull request with the plan.\n\n" + | ||
| "If multiple root causes were identified, use --cause to specify which one.\n\n" + |
There was a problem hiding this comment.
The command docs claim it will "generate code changes and create a pull request", but the implementation only triggers the "solution" step and outputs the solution artifact (no PR URL/number is surfaced). Either adjust the documentation to match current behavior (generate a plan) or implement/trigger the PR creation step and print the resulting PR info.
| export function triggerRootCauseAnalysis( | ||
| orgSlug: string, | ||
| issueId: string | ||
| ): Promise<{ run_id: number }> { | ||
| return apiRequest<{ run_id: number }>( | ||
| `/organizations/${orgSlug}/issues/${issueId}/autofix/`, | ||
| { | ||
| method: "POST", | ||
| body: { step: "root_cause" }, | ||
| } | ||
| ); | ||
| } | ||
|
|
||
| /** | ||
| * Get the current autofix state for an issue. | ||
| * | ||
| * @param orgSlug - The organization slug | ||
| * @param issueId - The numeric Sentry issue ID | ||
| * @returns The autofix state, or null if no autofix has been run | ||
| */ | ||
| export async function getAutofixState( | ||
| orgSlug: string, | ||
| issueId: string | ||
| ): Promise<AutofixState | null> { | ||
| const response = await apiRequest<AutofixResponse>( | ||
| `/organizations/${orgSlug}/issues/${issueId}/autofix/` | ||
| ); | ||
|
|
||
| return response.autofix; | ||
| } |
There was a problem hiding this comment.
These new Autofix API helpers are not passing a Zod schema to apiRequest, unlike the existing high-level API methods in this file. Consider importing AutofixResponseSchema (and a small schema for the trigger response) and providing schema: so malformed responses fail fast and types stay trustworthy.
| const lines: string[] = []; | ||
|
|
||
| // Cause header | ||
| lines.push(`${yellow(`Cause #${index}`)}: ${cause.description}`); |
There was a problem hiding this comment.
formatRootCause labels causes as Cause #${index} even though each RootCause already has an id field. If the API returns non-sequential IDs or the list is filtered/reordered, the displayed ID won’t match the actual cause ID users should reference. Prefer displaying cause.id (and keep --cause selection consistent with that).
| lines.push(`${yellow(`Cause #${index}`)}: ${cause.description}`); | |
| const displayId = cause.id ?? index; | |
| lines.push(`${yellow(`Cause #${displayId}`)}: ${cause.description}`); |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Semver Impact of This PR🟡 Minor (new features) 📋 Changelog PreviewThis is how your changes will appear in the changelog. New Features ✨Issue
Other
Bug Fixes 🐛Issue
Other
Documentation 📚
Internal Changes 🔧
🤖 This preview updates automatically when you update the PR. |
betegon
changed the title
betegon
left a comment
betegon
left a comment
There was a problem hiding this comment.
Review Comments
Thanks for adding the Seer AI integration! The implementation looks solid overall. I have a few suggestions to improve consistency and maintainability:
1. Rename autofix → seer (naming consistency)
Files: src/types/autofix.ts, src/lib/formatters/autofix.ts, and corresponding test files
The term "autofix" is Sentry's internal API naming, but users interact with sentry issue explain and sentry issue plan — they never see "autofix" anywhere.
Please rename these files to use seer (the product name):
src/types/autofix.ts→src/types/seer.tssrc/lib/formatters/autofix.ts→src/lib/formatters/seer.tstest/types/autofix.test.ts→test/types/seer.test.tstest/lib/formatters/autofix.test.ts→test/lib/formatters/seer.test.ts
This makes the codebase more intuitive — when someone looks for "explain" or "plan" related code, seer.ts is more discoverable than autofix.ts.
2. Extract polling to src/lib/polling.ts
File: src/commands/issue/utils.ts (lines 111-163)
The pollAutofixState() function is generic enough to be reused by other commands. Please extract it to src/lib/polling.ts as a general utility.
This would allow future commands to reuse the same polling pattern with spinner animation.
3. Stale comment reference to "fix command"
File: src/types/autofix.ts:165
// Solution Artifact (from fix command)This should say "plan command" since the command was renamed from fix to plan.
4. JSDoc improvements needed
Several exported functions are missing @param and @returns documentation:
src/types/autofix.ts:
isTerminalStatus(line 268) — missing@param status,@returnsextractRootCauses(line 275) — missing@param state,@returns
src/lib/formatters/autofix.ts:
getSpinnerFrame(line 26) — missing@param tick,@returnsformatRootCauseHeader(line 161) — missing@returns
src/commands/issue/plan.ts:
validateCauseSelection(line 84) — has only brief description, missing@param,@returns,@throws
5. Consider --force flag for explain command
File: src/commands/issue/explain.ts
Currently, if an analysis already exists and is COMPLETED, the command returns cached results. It might make sense to add a --force flag that triggers a fresh analysis even when one exists.
This could be useful when:
- The issue has new events since the last analysis
- The user wants to retry after fixing code mappings
Example: sentry issue explain 123456 --force
Let me know if you have questions on any of these!
|
I'm also worried about the 10 minutes timeout on the pooling. isn't it too much? I mean for agents, imagine your claude code or codex waiting for 10mins (if they don't time out earlier). wdyt of a 3mins timeout or something? then we can say, try it again and the agent will do it again. maybe we could ask the seer team on the average root cause analysis duration |
|
yeah 3 mins seems reasonable, will change that |
|
one more @MathurAditya724 , we show this message when running
It'd be cool if we put something like: Starting root cause analysis, it can take several minutes The first time I run one it was instant as the issue had it already, and then run another one where it had to run and it's taking a bit, so that text could help on what to expect, both for users and agents. |
betegon
left a comment
betegon
left a comment
There was a problem hiding this comment.
Approving this so we can get it merged.
I'll take care of a follow up here to make explain and plan support short issue ids like the other commands do.
closes #42