Skip to content

feat(autofix): Autofix introspection analytics#115891

Merged
Zylphrex merged 2 commits into
masterfrom
txiao/feat/autofix-introspection-analytics
May 21, 2026
Merged

feat(autofix): Autofix introspection analytics#115891
Zylphrex merged 2 commits into
masterfrom
txiao/feat/autofix-introspection-analytics

Conversation

@Zylphrex
Copy link
Copy Markdown
Member

This adds an introspection after each autofix step that uses a LLM to judge if we should continue. For now this is behind a feature flag and will only emit analytics as we fine tune this.

This adds an introspection after each autofix step that uses a LLM to judge if
we should continue. For now this is behind a feature flag and will only emit
analytics as we fine tune this.
@Zylphrex Zylphrex requested review from a team as code owners May 20, 2026 15:43
@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label May 20, 2026
Comment thread src/sentry/seer/autofix/on_completion_hook.py Outdated
Comment on lines +422 to +425
prompt,
"introspection_decision_code_changes",
)
except Exception:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: The introspection logic performs a synchronous, 30-second blocking poll within an RPC handler, which can cause worker timeouts and stall the entire autofix pipeline.
Severity: HIGH

Suggested Fix

The introspection logic should be executed asynchronously to avoid blocking the web worker and the main autofix pipeline. Move the call to run_introspection into a background task (e.g., using Sentry's task queue) so that it does not delay or prevent the trigger_autofix_agent call from executing promptly.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: src/sentry/seer/autofix/introspection.py#L422-L425

Potential issue: The `run_introspection` function, called from the
`call_on_completion_hook` RPC handler, initiates a synchronous poll for up to 30 seconds
by calling `client.get_run(..., blocking=True, poll_timeout=30)`. This blocks a web
worker for a long duration, risking worker timeouts and process kills. Because this
blocking analytics call occurs before the `trigger_autofix_agent` function is called to
continue the pipeline, a timeout or failure during introspection will prevent the
autofix process from proceeding to the next step. This turns an analytics feature into a
potential blocker for the entire autofix product.

Also affects:

  • src/sentry/seer/autofix/on_completion_hook.py:344~352

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is fine for now

Copy link
Copy Markdown
Member

@JoshFerge JoshFerge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense to me! when ready we could add some evals for this as well!

@Zylphrex Zylphrex merged commit c038fa9 into master May 21, 2026
62 checks passed
@Zylphrex Zylphrex deleted the txiao/feat/autofix-introspection-analytics branch May 21, 2026 14:31
JonasBa pushed a commit that referenced this pull request May 21, 2026
This adds an introspection after each autofix step that uses a LLM to
judge if we should continue. For now this is behind a feature flag and
will only emit analytics as we fine tune this.
natemoo-re pushed a commit that referenced this pull request May 21, 2026
This adds an introspection after each autofix step that uses a LLM to
judge if we should continue. For now this is behind a feature flag and
will only emit analytics as we fine tune this.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants