Add RFC-0013: Agent Improvement Workflow - #42
Open
Nehanth wants to merge 2 commits into
Open
Conversation
An Improve tab that turns issue detection findings into reviewed fixes through a pluggable acquire/apply/submit pipeline, building on RFC-0012's trace events. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 25, 2026
Per review discussion: enterprise teams should be able to pull fix prompts programmatically into their own managed coding agents, not only via the UI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jeremyeder
reviewed
Aug 27, 2026
| **2. Apply a fix.** How a diagnosis becomes file changes in that checkout. Backends: OpenCode (the bundled, MIT-licensed default), Claude Code, other coding agents, or a plain script for mechanical fixes with no model calls at all. The backend is invoked in the workspace with the diagnosis and trace evidence as input, edits files, and exits. The only secret this phase ever sees is its own model access (an AI Gateway endpoint) — the interfaces on either side hold the repository credentials, so a coding agent structurally cannot touch them. | ||
|
|
||
| **3. Submit for human review.** How the changed code reaches people. Backends: a GitHub pull request (default), a GitLab merge request, a patch bundle, a pushed branch for teams with their own review tooling, or delegating to a forge's own coding agent (for example, filing a GitHub issue assigned to Copilot). **Copy prompt** is the simplest backend of all: no checkout and no branch — MLflow hands the user the diagnosis and evidence as a ready-made prompt for whatever coding agent they drive themselves, skipping phases 1 and 2 entirely. | ||
| **3. Submit for human review.** How the changed code reaches people. Backends: a GitHub pull request (default), a GitLab merge request, a patch bundle, a pushed branch for teams with their own review tooling, or delegating to a forge's own coding agent (for example, filing a GitHub issue assigned to Copilot). **Copy prompt** is the simplest backend of all: no checkout and no branch — MLflow hands the user the diagnosis and evidence as a ready-made prompt for whatever coding agent they drive themselves, skipping phases 1 and 2 entirely. Copy prompt is available both as a UI action and programmatically — an API returns the fix prompt for a diagnosis, so enterprise teams can feed it into their own managed coding agents or software-factory pipelines without going through the UI. |
There was a problem hiding this comment.
Yep - let us drive MLflow "headless". The analysis/prompt "work package" gets fed back to our systems, so it follows the blessed flows (heavy investment - need+want to keep that control). Thank @Nehanth.
B-Step62
reviewed
Sep 1, 2026
| import mlflow | ||
|
|
||
| # One-time setup: connect the agent's repository to the experiment | ||
| mlflow.genai.connect_repo( |
Contributor
There was a problem hiding this comment.
nit: I think this should be platform-level capability, not GenAI
| A support agent's system prompt was written before the product's new billing system launched. Users now ask billing questions the prompt gives no guidance on, and the agent answers vaguely. Nothing errors — but the relevance and completeness scores on those conversations drop, because the judge can see from the conversation alone that the answers are not addressing what users asked. | ||
|
|
||
| 1. The threshold event fires on the rolling average and triggers the workflow. The diagnosis points at the prompt rather than the code, with the low-scoring conversations as evidence. | ||
| 2. Because MLflow already has a prompt registry and [prompt optimization](https://mlflow.org/docs/latest/genai/prompt-registry/optimize-prompts/), the fix path needs no coding agent: the low-scoring traces become the evaluation set, `optimize_prompts()` rewrites the prompt against them, and the improved version is registered with before-and-after scores shown in the diagnosis. |
Contributor
There was a problem hiding this comment.
nit: Prompt optimization generally requires non-trivial number of ground truth or target function. I would let the harness directly update prompt instead of relying on optimization here.
| # runs when RFC-0012's events fire; list the open ones and apply one | ||
| suggestions = mlflow.genai.list_improvements(experiment_id="12345", status="open") | ||
| job = mlflow.genai.apply_improvement( | ||
| suggestion_id=suggestions[0].id, |
Contributor
There was a problem hiding this comment.
- Harness execution should be a part of Assistant, because that will be the main interface for UI users to interact with harnesses. We shouldn't introduce different execution mechanism between UI and SDK.
- With (1), this could be more generic API that trigger harness with a single prompt. It allows harnesses to handle suggestions with more flexibility, e.g., combine multiple suggestions together.
|
|
||
| A team runs a support agent traced to MLflow. A provider deprecates the model identifier the agent has hardcoded, and every request starts failing. | ||
|
|
||
| 1. In the experiment's **Improve** tab, the team has already connected their repository: URL and an access token, stored through MLflow's existing secret handling, plus the harness choice. This is one-time setup. The connection lives at the experiment level, since an experiment usually tracks a single agent — a team running several agents connects each experiment to its own repository. |
Contributor
There was a problem hiding this comment.
We are adding Jira-like "Issues" tab in the UI. The workflow can start from there.
|
|
||
| 6. On merge, the issue is resolved in MLflow, and the traces that exposed the failure are added to an evaluation dataset ([`merge_records()`](https://mlflow.org/docs/latest/genai/datasets/) accepts traces directly) — the failure is now a regression test. A pull request closed without merging means the reviewer rejected the change: the diagnosis is marked **rejected**, and later detection runs do not re-propose the same fix. | ||
|
|
||
| MLflow learns the outcome by polling the pull request through the submission backend, with backoff — checks start frequent and stretch out as the PR ages. After a bounded window, MLflow stops polling altogether and marks the diagnosis **stale**; opening it in the tab re-checks once, on demand. A PR that humans ignore therefore costs nothing in the background. Whether the forge should instead push events to MLflow is left to the detailed design: that would be a new inbound endpoint on the MLflow server (RFC-0012's webhooks only send outbound), which is a larger security decision than polling. |
Contributor
There was a problem hiding this comment.
Not sure if this is the right mechanism. Github REST API has strict rate limit and a server can keep polling indefinite amount of times if some stale issue is not cleaned up. I prefer letting users to set up Github action that invokes MLflow APIs.
|
|
||
| The same shortcut extends beyond prompts as the registries land: a skill is, like a prompt, mostly instructions as text, so a diagnosis rooted in a skill's instructions can become a new skill version registered through the Skill Registry and rolled out or back through the registry's own versioning. The coding harness is reserved for what actually is code. | ||
|
|
||
| ### Keep detection running on a schedule |
Contributor
There was a problem hiding this comment.
This has been worked on in databricks side and we can port the same design (e.g. checkpointing) to OSS. Let's focus on the improvement workflow itself in this RFC.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The second half of the #38 split, building on RFC-0012 (#41): an Improve tab where issue detection findings are pinned to the connected repository and a coding agent produces a fix as a pull request, through a pluggable pipeline (acquire source → apply a fix → submit for review) with OpenCode bundled as the open-source default. Nothing merges without human review, and fixed failures become regression datasets.
Same template and journeys-first structure; review feedback from #38 is incorporated.