Should evidence-backed agent change proposals exist?

[vittorfp]

I am looking for early feedback on Agent Change Proposals.

The core idea:

Existing observability and eval tools show what happened. ACP tries to describe what should change, why, and how to validate it.

The artifact I most want feedback on is change_proposal:

• Is this useful as a review/CI artifact for agent behavior changes?
• Which fields are missing, confusing, or too speculative?
• Would this belong outside existing eval/observability tools, or should it live inside them?
• What would make a proposal trustworthy enough to review?

Useful links:

• README: https://github.com/vittorfp/agent-change-proposals
• First proposal walkthrough: https://github.com/vittorfp/agent-change-proposals/blob/main/docs/first-proposal.md
• Positioning: https://github.com/vittorfp/agent-change-proposals/blob/main/docs/not-an-observability-platform.md
• Change proposal schema: https://github.com/vittorfp/agent-change-proposals/blob/main/schemas/change_proposal.schema.json
• Example proposal: https://github.com/vittorfp/agent-change-proposals/blob/main/examples/rag-missed-retrieval/change_proposal.example.json

I am especially interested in feedback from people running agent evals, observability pipelines, or internal agent platforms.

[vittorfp]

Concrete feedback ask after v0.5.0:

The current best example is the declared-intent proposal:
https://github.com/vittorfp/agent-change-proposals/blob/main/examples/declared-intent/change_proposal.example.json

Latest release:
https://github.com/vittorfp/agent-change-proposals/releases/tag/v0.5.0

I am especially looking for workflow-level reactions from people running agent evals, observability pipelines, or internal agent platforms:

1. Would this artifact fit PR review, CI, eval review, incident follow-up, or none of those?
2. Which fields are missing, confusing, or too speculative?
3. What evidence would make a proposal trustworthy enough to review?
4. Which existing tool or process already solves this for you?
5. Which integration should be made more faithful first: OpenInference/Phoenix, Langfuse, LangSmith, Braintrust, or plain JSON traces?

I started a feedback sprint here and will summarize direct reactions back into the repo:
https://github.com/vittorfp/agent-change-proposals/blob/main/docs/feedback-sprint-2026-10-04.md

[GioiaZheng]

Useful for eval review and incident follow-up, but I would not use the current schema as a CI gate yet. change_proposal.schema.json only requires nested values to be objects or arrays, so an underspecified proposal still validates. Make validation machine-readable: immutable evidence refs, baseline/candidate artifact IDs, evaluator and dataset versions, metric threshold, sample count, and rollback trigger. I would stabilize plain JSON before adding platform integrations.