-
Notifications
You must be signed in to change notification settings - Fork 4
From human override to learning signal
A human edit becomes useful improvement data only when the original proposal, accepted change, rationale and decision context remain connected.
Human overrides can reveal where an agent repeatedly misunderstands evidence, policy, risk or domain practice. Most applications keep only the final corrected output, which erases the intervention. CHAP's decide.override preserves the proposal-to-decision relationship through a structured diff, rationale, tags and an intent_preserved classification.
That structure makes overrides candidates for evaluation and learning. It does not make them automatically safe or suitable for model training.
An override occurs when a reviewer accepts a modified version of an agent's output rather than approving it as-is or rejecting it completely.
Examples include:
- Changing the severity of a code-review comment.
- Removing an unsupported claim from a report.
- Reducing the amount of a proposed refund.
- Replacing the recommended supplier.
- Correcting a medication dose before approving a summary.
CHAP can distinguish a refining override, where the original intent is preserved, from a substituting override, where the reviewer changes the core direction. That distinction helps analysts avoid treating a spelling correction and a reversed recommendation as equivalent feedback.
| Element | Why it matters |
|---|---|
| Original artefact | Establishes what the agent actually proposed |
| Accepted artefact or patch | Shows the intervention rather than only the final state |
| Reviewer and authority | Provides context for how much weight to give the decision |
| Rationale | Explains the declared reason without requiring private chain of thought |
| Tags | Supports aggregation across many decisions |
intent_preserved |
Separates refinement from substitution |
| Agent and system version | Lets teams identify regressions and compare releases |
| Citations and policy references | Connects the correction to supporting evidence or rules |
| Operational mode | Distinguishes shadow, trial and production behaviour |
Without this context, a dataset may teach the model to imitate a local edit without understanding when the edit applies.
Record the proposal, final accepted artefact and explicit decision evidence. Avoid capturing every cursor movement, private deliberation or irrelevant screen activity.
Verify that the override refers to the expected task and artefact, the participant was authorised, and the sequence has not been altered. Exclude incomplete or reconciled records until their state is clear.
Convert patches into features that match the domain. A JSON field replacement, deleted paragraph and changed tool argument may need different analysis. Keep access to the original patch for auditability.
Overrides may contain customer data, health information, source code, legal advice or confidential business rules. Apply retention, access, consent, de-identification and contractual controls before reuse.
Separate low-risk stylistic refinement from high-consequence policy substitution. Segment by agent version, task type, reviewer role, organisation, operational mode and applicable policy.
Human decisions can be wrong or inconsistent. Sample overrides for second review, measure inter-reviewer disagreement where appropriate, and flag decisions made under exceptional or emergency conditions.
Not every pattern requires model training. A repeated override may be better addressed by:
- Improving the prompt or task specification.
- Adding missing context through MCP.
- Changing a tool schema or validation rule.
- Updating a policy or reviewer interface.
- Creating a deterministic guardrail.
- Building an evaluation set.
- Fine-tuning or preference optimisation after stronger validation.
Hold out representative cases and test whether the change reduces the target error without creating new failures. Compare unchanged approvals, overrides, rejections and escalations, not just overall task success.
Track whether the override pattern declines for the intended agent version and task segment. A falling override rate is not automatically good if reviewers have become less attentive or the interface has made correction harder.
| Measure | Interpretation | Caution |
|---|---|---|
| Override rate | Share of reviewed proposals accepted after modification | Depends on task difficulty and review policy |
| Substitution rate | Share of overrides with intent not preserved | Requires consistent classification |
| Override tag frequency | Common declared failure modes | Tags need governance and reviewer guidance |
| Patch magnitude | Approximate extent of change | Large text diffs can still be semantically minor |
| Reviewer disagreement | Variation across qualified reviewers | May reflect legitimate policy ambiguity |
| Post-change recurrence | Whether a targeted error returns after improvement | Control for agent and task mix changes |
Do not rank individual reviewers by raw override rate. Reviewers assigned harder cases will appear more intervention-heavy, and pressure to reduce the metric can suppress necessary corrections.
Suppose reviewers repeatedly remove sentences that are not supported by cited sources and tag the override unsupported-claim.
The first intervention might justify a prompt update requiring citation coverage. A larger pattern may justify an evaluation set that checks whether each material claim has a supporting citation. Only after measuring the failure across versions should the team decide whether training is the right intervention.
The CHAP record helps reconstruct each proposal, edit, reason and agent version. Domain reviewers still decide whether the correction itself is valid.
Overrides are also useful when no model is updated. They can show where policy is unclear, where a tool lacks necessary fields, where one team bears excessive review burden or where agents are being used outside their intended scope.
This is why it is better to call them learning signals rather than training examples. The organisation may need to learn before the model does.
- Why AI audit logs do not capture human judgement
- CHAP with LangGraph human-in-the-loop
- What CHAP proves, and does not prove, for AI governance
If you are already collecting human edits, star CHAP and share which override fields would make them usable as governed learning signals.
Last reviewed: 25 August 2026.