Should autonomous agents share a common evidence contract for external outcomes?

[AyobamiH]

I've published four retrospective engineering case studies from separate working systems. They raise a question I want other engineers to challenge before building any new integration:

When an autonomous agent says that an external operation is finished, what evidence should another independent tool be able to inspect?

Four existing implementations — not one proposed framework

Independent work	Observed result	Important limit
Post-Once: Thirteen deferred vaults were not thirteen lock collisions	The original 13/18 deferral consisted of two observed lock refusals plus eleven policy-driven skips. Repair and independent observer evidence subsequently covered two adjacent natural 18/18 successful collection cycles (release record).	Collector/durability acceptance is not blanket provider, editorial, or business-outcome acceptance.
AgentProof: What seventeen production checks proved	A real immutable OIDC identity mismatch was rejected, then corrected; a pinned disposable repository-patch canary passed 17/17 hosted checks, with receipts and successor-chain validation (acceptance).	This does not establish generic exactly-once provider effects, HSM custody, or permission to publish releases.
OpsTruth: The verifier rejected the shape of success	A malformed numeric GitHub repository identity was rejected at an independent verification boundary. The adapter was corrected without loosening the versioned subject-binding contract (repair).	GitHub CI evidence cannot prove unrelated production/business effects; OpsTruth remains read-only.
DoneState: A pull request was not the finish line	Bounded PR-only autonomous engineering ran against exact heads; a separately pinned OpsTruth verifier provided the completion decision (verified successor, selected-App acceptance).	Reviewable PR, green checks, independently verified bounded objective and customer usefulness are different claims. No automatic merge authority.

These are dated, scoped, author-published case studies. Failures are retained alongside successful evidence. AI assistance was involved in implementation and writing; the posts are not independent peer-reviewed evaluations.

Questions for maintainers and practitioners

1. What minimal interoperable record would help a read-only verifier reason about an external effect without trusting the agent's own tool response or granting it write authority?
2. Which claims should remain deliberately separate: authorisation, attempted action, provider-committed effect, observed postcondition, signed receipt, and usefulness to the requester?
3. How would you represent an effect that may still commit after its acknowledgement is lost? When does safe reconciliation become impossible without an external service's idempotency or readback guarantees?
4. Is a portable signed receipt useful on its own, or would existing GitHub artefacts, OpenTelemetry conventions, provider idempotency and clear documentation already solve the practical need?
5. Can you show a concrete failure, prior art, open standard, simpler implementation or counterexample that makes cross-tool interoperability unnecessary or materially different?

Decision rule and non-goals

Each project remains independent. No shared runtime, fleet coordinator, new database, package, authority expansion or product merger is approved by opening this discussion. A small versioned adapter or common evidence vocabulary is worth considering only when external examples show that it solves a real problem that existing tools do not.

Useful responses include links to reproducible traces, provider semantics, adversarial tests, independent verifiers, and comparisons with prior work. Please avoid credentials, customer data or unredacted private execution logs.

The goal is to learn what deserves to be standardised—including the possibility that nothing needs to be combined.

Public-safe reproduction: Post-Once lock-deferral policy (2 November 2026)

The original private operational incident had 18 enabled vaults and 13 reported deferrals, but only two actual RunnerBusy lock refusals; eleven later vaults were not attempted due to the consecutive-busy safety policy. A public, credential-free JavaScript model with eight tests now reproduces that classification mechanism. The merged, governed source passed 48 governance and 34 application tests on each of two owner-controlled machines.

This is a synthetic behavioural model and not an anonymous copy of the private provider receipts, a rerun of October production effects, third-party adoption, or a new shared runtime. Please challenge the distinction between a refused attempt, policy-skipped work and lost observations. The original Sites Git synchronisation is separately tracked in issue #47.

[AyobamiH]

Runnable challenge: what counts as an independently verified external effect?

I turned the earlier design question into a small, public, falsifiable experiment rather than another integration roadmap:

• Reproduction instructions and eight-case fault matrix
• Standards-first comparison — in-toto/DSSE, Sigstore, OTel, W3C PROV, provider GET and the 2026-10-22 HTTPAPI Idempotency-Key draft
• Open research/code review PR #29 and open limitations/acceptance register

The source-only fixture uses no GitHub release publication, credentials or provider calls. It reuses existing AgentProof release-v1 tests and pins the reviewed source hashes. Two owner-controlled machines have run the exact PR commit (6687b66a772ba8db3831c4f09f692a2e5e8fa7b3), yielding 8/8 passing synthetic fault cases. A changed test source intentionally returns BLOCKED: SOURCE_DRIFT. Neither machine counts as an independent external reproduction.

git clone https://github.com/AyobamiH/agentproof.git
cd agentproof
git fetch origin research/repro-outcome-uncertainty-20261101
git checkout --detach 6687b66a772ba8db3831c4f09f692a2e5e8fa7b3
npm ci --ignore-scripts --no-audit --no-fund
npm run build
node scripts/reproduce-release-uncertainty.mjs

An important deliberately negative assertion: a trusted signed Receipt V2 still has independentOutcomeVerified=false; a signature does not turn a claimed external effect into independent proof. The real historical GitHub release is fenced and must not be republished.

Three precise questions for outside reviewers:

1. Can an in-toto Statement + DSSE envelope, a correctly scoped provider GET and a trustworthy observer predicate already express the entire decision? If so, show the minimal representation and tell us not to build another standard.
2. If the provider may have committed but GET/readback is unavailable or eventually consistent, what should the consumer decide without another write? Can you demonstrate a counterexample?
3. Which trust/authority/freshness/identity binding must be verified in addition to the cryptographic signature, and what existing format already captures it?

Real reproductions with pinned commits, redacted outputs and counterexamples are preferable to a favourable opinion. This is a review of existing bounded code, not a solicitation to merge the products.

Prior-art refinement (2 November 2026)

The standards-first check found existing work closer to this problem than a general new framework: in-toto Release predicate v0.2 binds release identities to artifact hashes under registry authority; in-toto agent-action PR #588 proposes signed, intermediary-witnessed tool-call evidence; and OpenTelemetry GenAI issue #470 was closed as not planned rather than standardising an evidence-artifact link without demonstrated mainstream use.

These cover different claims. A gateway-witnessed attempt is not proof of provider-side commitment when the acknowledgement is lost. I asked a bounded question on PR #588 about how its observation model represents that uncertainty. The independent reproduction request accepts failure cases or a minimal standards-based solution. No outside result has been verified yet; no new predicate has been requested.