Skip to content

feat: emit incident intent artifacts for the Agent Manager remediation loop #5

Description

@AllenMuu

Context

StackWatch already has an issue focused on strengthening deterministic error identity and history (#3). The next AI-native SDLC step should not be automatic remediation yet.

The immediate goal is to turn a high-confidence production incident/RCA into a portable intent artifact that another coding agent workflow can consume safely.

Target loop:

Production Error
  -> StackWatch grouping / RCA
  -> confidence + risk gate
  -> intent artifact
  -> Agent Manager
  -> planner
  -> implementer
  -> verifier
  -> human approval / PR

StackWatch should own the production incident facts and RCA evidence. Agent Manager should own the downstream development workflow.


Goal

Add an explicit export boundary from StackWatch into an AI-native SDLC workflow by generating a versioned intent artifact from an analyzed incident.

The first version should be human-reviewed and non-autonomous:

  • no automatic code modification;
  • no automatic PR creation;
  • no direct remediation execution;
  • no hidden handoff through raw chat history.

The output should be structured enough that Agent Manager can consume it without depending on StackWatch internals.


P0 — Define IncidentIntent mapping

Create a dedicated application service that maps StackWatch domain data into the cross-agent intent schema.

Conceptually:

ErrorGroup
+ ErrorOccurrence samples
+ RCA result
+ confidence
+ evidence
+ environment / release context
        |
        v
IncidentIntentBuilder
        |
        v
intent.yaml / intent.json

Suggested output:

version: v1
kind: intent
id: incident-<group-id>-<timestamp>
source:
  system: stackwatch
  type: production-incident
incident:
  group_id: ...
  app_name: order-service
  environment: prod
  first_seen: ...
  last_seen: ...
  occurrence_count: 42
  release: ...
summary: Redis timeout causes order submission failures
problem: ...
goals:
  - identify the concrete code/configuration cause
  - implement the smallest safe fix
non_goals:
  - unrelated refactors
constraints:
  - preserve backward compatibility
acceptance_criteria:
  - existing tests pass
  - regression test covers the failure mode
risk_level: medium
rca:
  summary: ...
  confidence: 0.91
  suspected_components:
    - ...
  evidence:
    - type: stack-frame
      value: ...
    - type: trace
      value: ...
links:
  trace_ids: []
  related_groups: []

The exact schema should align with Agent Manager's artifact protocol once available. Until then, keep the StackWatch mapper isolated behind a small adapter so fields can evolve without contaminating core RCA models.


P0 — Separate facts, inference and requested work

The exported artifact must make these categories explicit:

Facts

Directly observed production data:

  • exception/root-cause type;
  • normalized message;
  • application frames;
  • occurrence count;
  • first/last seen;
  • app/environment/release;
  • trace/span IDs;
  • relevant sampled occurrences.

Inference

LLM/RCA-derived conclusions:

  • suspected root cause;
  • suspected component/file/module;
  • confidence;
  • alternative hypotheses;
  • missing evidence.

Requested work

What a downstream development agent should actually do:

  • goal;
  • constraints;
  • non-goals;
  • acceptance criteria;
  • verification expectations.

Do not present an LLM hypothesis as an observed fact.


P0 — Confidence and risk gate

Introduce an explicit gate before intent generation is considered actionable.

Suggested model:

record RemediationReadiness(
    double rcaConfidence,
    RiskLevel riskLevel,
    boolean enoughEvidence,
    List<String> blockers
) {}

Example policy:

confidence >= threshold
AND enough evidence
AND risk is not prohibited
  -> exportable as "ready-for-planning"
else
  -> exportable only as "needs-investigation"

Requirements

  • Confidence threshold is configurable.
  • High-risk incident classes can never be silently promoted to remediation-ready.
  • Missing evidence is represented explicitly.
  • The artifact includes readiness status and blockers.
  • A user can still export a low-confidence incident for investigation, but it must not be labeled as remediation-ready.

P0 — Add export API/CLI boundary

Provide at least one stable export path.

Possible options:

POST /api/error-groups/{id}/intent
GET  /api/error-groups/{id}/intent

or a CLI/module command if StackWatch already has an appropriate command surface.

Preferred first behavior:

error group / analysis
  -> generate preview
  -> user reviews
  -> export YAML/JSON

Requirements

  • Preview does not mutate production state.
  • Export is deterministic for the same analysis/version where possible.
  • Output can be serialized as JSON and YAML.
  • Schema version is always present.
  • StackWatch-specific database entities are not leaked as required downstream dependencies.

P0 — Evidence packaging

Do not dump full raw logs or unlimited traces into the artifact.

Add a bounded evidence model.

Suggested types:

stack-frame
exception
normalized-message
trace-id
span-id
release
metric
log-snippet
related-error-group

Rules:

  • preserve identifiers/links needed for drill-down;
  • include only representative sampled evidence;
  • cap evidence count/size;
  • redact configured sensitive values;
  • keep raw large payloads outside the intent artifact.

P1 — Agent Manager integration adapter

Once Agent Manager's artifact protocol exists, implement a thin compatibility adapter.

Target:

StackWatch IncidentIntent
        |
        v
AgentManagerArtifactAdapter
        |
        v
agent-manager task import intent.yaml

Do not make StackWatch depend on Agent Manager runtime libraries if a schema/module boundary is sufficient.

Preferred dependency direction:

shared artifact contract/schema
       ^              ^
       |              |
 StackWatch      Agent Manager

If a shared package is premature, duplicate only the external contract and add compatibility tests/fixtures.


P1 — Verification hints from incident evidence

The exported intent should help the downstream verifier reproduce the issue.

Potential fields:

verification_hints:
  regression_test:
    expected_failure_before_fix: ...
    expected_success_after_fix: ...
  affected_paths: []
  relevant_config: []
  reproduction_notes: []

These are hints, not executable commands from the LLM.

Requirements

  • No untrusted production text is automatically executed as a shell command.
  • Reproduction steps are treated as data until reviewed/translated by the downstream workflow.
  • A downstream verifier can determine what evidence should prove the incident is fixed.

P1 — Lifecycle/status tracking

Add a minimal handoff status to the incident/group level if it fits the existing domain model.

Suggested states:

not_analyzed
analyzed
needs_investigation
ready_for_planning
intent_exported

Do not add implementation/PR deployment states to StackWatch unless there is a concrete integration need. Agent Manager/GitHub should remain authoritative for downstream engineering execution.


P2 — Closed-loop feedback

After the one-way export is stable, consider importing the eventual verification/remediation outcome back into StackWatch.

Future flow:

StackWatch incident
  -> intent
  -> Agent Manager workflow
  -> verification / PR
  -> deploy
  -> observe production
  -> resolution feedback

Possible future feedback:

  • PR URL;
  • deployed release;
  • verification artifact;
  • whether occurrence rate dropped;
  • resolved/reopened status;
  • lesson/memory candidate.

This is explicitly out of scope for the first implementation.


Security / safety requirements

  • Never execute log/exception text as commands.
  • Treat production payloads as untrusted input.
  • Redact configured sensitive data before export.
  • Keep human approval between generated intent and code-changing execution in the first version.
  • Do not automatically create/merge PRs.
  • Do not label low-confidence RCA as confirmed root cause.
  • Preserve audit metadata showing which analysis/version generated the artifact.

Suggested implementation order

1. Define external IncidentIntent DTO/schema
2. Define facts vs inference vs requested-work mapping
3. Implement IncidentIntentBuilder
4. Add readiness/confidence/risk policy
5. Add bounded Evidence model
6. Add JSON/YAML serialization + schema version
7. Add preview/export API
8. Add fixtures/contract tests
9. Add Agent Manager compatibility adapter after its artifact schema lands
10. Add verification hints

Acceptance criteria

  • A persisted/analyzed error group can produce a versioned intent artifact.
  • The artifact clearly separates observed facts from RCA inference.
  • RCA confidence and remediation readiness are explicit.
  • Low-confidence cases are marked needs-investigation rather than silently promoted.
  • The artifact contains enough bounded evidence for a downstream planner to understand the incident.
  • The artifact includes goals, constraints, non-goals and acceptance criteria.
  • JSON/YAML outputs are covered by schema/fixture tests.
  • Production text is never treated as executable instructions.
  • StackWatch remains usable without Agent Manager installed.
  • Once Agent Manager's artifact protocol is available, at least one compatibility test proves the exported intent can be imported/consumed.

Definition of done

Demonstrate this one-way workflow:

real/synthetic ErrorEvent
  -> grouping + RCA
  -> readiness evaluation
  -> intent artifact preview
  -> export
  -> Agent Manager task import / planner consumption

No automatic remediation is required. The purpose of this issue is to make Production → RCA → Intent a stable, auditable boundary so a safe AI-native SDLC loop can be built on top of it.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions