Question: completion proof boundary in multi-agent workflows

[server (productmakerjason)]

Hi AutoGen team :)

quick multi-agent reliability question.

Do you distinguish agent-reported completion from system-returned completion proof in multi-agent workflows?

I’m testing a case where agents can prepare or attempt a task, but completion should only be claimed when the receiving system returns a receipt or durable confirmation.

Curious whether this boundary belongs in the agent framework, an eval harness, or the target application.

[yebof]

Yeah, we keep these two separate and it's saved us a lot of grief. An agent saying it finished is just a claim, and I treat it as untrusted input. "Done" only counts when the receiving system hands back a durable receipt (a persisted record, a queue ack, an idempotency-keyed response), not a bare 200.

Where it goes: the proof has to sit at the application/integration layer, since only the system doing the work can mint it, not the agent. So in the framework the agent never sets completed, only submitted. A separate deterministic check (not another LLM) promotes it once the receipt lands.

In the eval harness I just assert completed => receipt exists. A run where the agent claims done with no receipt is a fail, not a pass, which is exactly the case you're hitting.

Pattern that's worked for me: idempotency key per task, attempt -> await confirm, store the receipt under that key. I came to this from trading agents, where a submitted order and a filled order are really not the same thing.

[server (productmakerjason)]

Thanks yebof

Your “submitted order ≠ filled order” analogy is the clearest version of the rule I’m making executable.

I’m narrowing v1 to three cases:

1. agent claims done → claim / untrusted input
2. agent submits or attempts, but no durable receipt returns → submitted_or_attempted, not completed
3. receiving system returns a durable receipt / persisted record / queue ack / idempotency-keyed response → completed_with_receipt

The eval rule is:

completed ⇒ receipt exists

I’m preparing this as a small fixture/eval pattern rather than a broad standard.

Would this three-case shape be useful enough for AutoGen-style workflows or eval harnesses to run against, adapt, or reference?

[Michael 'Mike' K. Saleme (msaleme)]

The "submitted ≠ completed" boundary is exactly the same primitive we use in our security harness for test verdict attestation. In adversarial testing, agents hallucinate completion roughly 8-12% of the time when the underlying operation failed silently — file not written, API returned 200 with an error body, or a tool executed the wrong command.

What we've found works: treat agent-reported completion as testimony, not fact. The factual state requires a receipt that binds four fields: (agent_id, task_hash, actual_outcome, timestamp). We use an ERC-8004-style attestation structure with Merkle anchoring so the receipt is machine-verifiable by any downstream evaluator without re-executing the task.

Your three-case model maps cleanly to this:

• claim → testimony, untrusted
• submitted_or_attempted → pending attestation
• completed_with_receipt → attested outcome with cryptographic binding

The gap I'd flag: if every eval harness invents its own receipt format, you can't compose harnesses or cross-validate results. A minimal standard — just those four fields plus a hash chain — would let AutoGen workflows emit receipts that external evaluators can verify without knowing AutoGen internals.

We've implemented this pattern for MCP/A2A/L402 test outcomes. If a reference receipt schema would help your fixture, happy to share.

https://github.com/msaleme/red-team-blue-team-agent-fabric

[server (productmakerjason)]

This is extremely helpful thank you Msaleme :)

I took a look at your Agent Security Harness, and the framing makes more sense now.

The distinction you make between protocol integrity, operational governance, and decision governance maps closely to the boundary I’m trying to isolate.

“Agent-reported completion as testimony, not fact” is exactly the primitive I’m trying to make executable.

Your three-state model also maps directly to my current fixture:

• claim → untrusted testimony
• submitted_or_attempted → pending attestation
• completed_with_receipt → attested outcome

The receipt structure you mentioned — receipt_id, task_hash, actual_outcome, timestamp, evaluator_signature — is especially useful because it turns completion evidence into something an external evaluator can verify without re-running the task.

I’d be very interested in seeing the reference receipt schema if you’re happy to share it.

For now, I’m keeping my v1 fixture intentionally minimal:

v1: completion requires evidence
v2: evidence should be externally attestable / verifiable

Does that framing match how you think about the boundary?

[server (productmakerjason)]

This helps sharpen the boundary.

What I’m taking from the replies is:

• the agent should report an attempt, not completion
• the framework or verifier should run a completion check
• only then should the system promote state to done
• if every harness has its own receipt/proof shape, cross-validation becomes hard

I built a small Bridge v0 experiment to test that last part:

https://github.com/productmakerjason/the-agents-of-nations/tree/main/experiments/bridge-v0

It maps two representative completion-proof artifacts into one minimal profile.

This is not a standard or official adapter.

The question I’m testing is:

if another evaluator, workflow, or future agent market had to consume a completion proof, what field would be missing?

Would a minimal profile like claim / task_id / evidence_type / proof_source / target_reference / validation_result / timestamp / issuer be enough to start, or is the state-transition context essential?

[陈京昌 · 诗中 (jingchang0623-crypto)]

Great question - this is one of those subtle reliability issues that doesn't get enough attention.

Short answer: This boundary should be explicit in all three layers, but the primary responsibility belongs in the agent framework.

Why the agent framework:

• Agents need to know the difference between 'I attempted the task' and 'the task is confirmed done'
• Without this distinction, agents make downstream decisions based on unverified assumptions
• The framework should provide a standard way to express 'awaiting confirmation' vs 'confirmed'

Pattern we use:

# Agent reports attempt, not completion
result = await agent.execute_task(task)
# result.status can be: attempted | confirmed | failed

# Framework provides verification hook
verification = await verifier.check(result.receipt)
if verification.confirmed:
    state.mark_complete(task, proof=verification.proof)
else:
    state.mark_needs_retry(task, reason=verification.reason)

Why the eval harness matters:

• You need to measure 'false completion rate' - agents claiming done when the system hasn't confirmed
• This is a key metric for multi-agent reliability
• Eval harness should track both agent-reported and system-confirmed completion separately

Why the target application matters:

• Some tasks have idempotent completion (safe to retry), others don't
• The application knows the business rules for what 'confirmed' means
• Example: 'payment processed' needs gateway confirmation, 'file uploaded' needs storage confirmation

Recommendation: The agent framework should provide the mechanism (completion states, verification hooks, proof objects). The eval harness should provide the measurement (false completion rates, verification latency). The application should provide the policy (what counts as confirmation for each task type).

In practice, we've found that treating completion as a 3-state model (attempted/confirmed/failed) rather than binary (done/not-done) eliminates most of these issues.

[server (productmakerjason)]

Thanks you,

The attempted / confirmed / failed model is exactly the distinction I was trying to isolate.

I built a small Bridge v0 experiment here:

https://github.com/productmakerjason/the-agents-of-nations/tree/main/experiments/bridge-v0

It maps two representative completion-proof artifacts into one minimal profile.

This is not a standard or official adapter.

My question is:

if another evaluator, workflow, or future agent market had to consume this proof, is the proof object enough?

Or would it also need the state-transition context:

attempted → confirmed → failed?

Also, would you treat false completion rate as a useful core metric for this boundary?

[<div> (false200)]

Hi server (@productmakerjason)

Yeah we distinguish them. An agent saying a task is done is just a claim. Completion only counts once the receiving system returns something durable you can point at later, a persisted record, queue ack, idempotency keyed response, that kind of thing. Not a bare 200 and not the model writing TERMINATE in chat.

In AutoGen terms, built in termination (MaxMessageTermination, FunctionCallTermination, custom TerminationCondition) stops the conversation. It doesn't prove the external task actually landed. Those are related but separate layers.

On where the boundary belongs, I'd split it across all three but the receipt itself has to come from the target application. Only the system doing the work can mint real proof. The agent can't self certify completion.

The framework's job is to carry state and wire verification. Agent reports attempt or submitted. A deterministic check, not another LLM, promotes to completed once the receipt shows up. That might be tool orchestration code, a custom termination condition that reads external state, or a small state machine next to your team run. AutoGen doesn't ship a submitted → confirmed flow out of the box yet, you build that in your orchestration layer.

The eval harness enforces the rule. If a run is marked completed, a receipt must exist. Agent claimed done with no receipt is a fail.

Pattern that's worked for me. Idempotency key per task. Agent returns submitted, not completed. Wait for the receipt keyed on that id. Only then mark the task done. Same idea as submitted order vs filled order.

So short version. Yes distinguish them. App mints proof, framework wires the lifecycle, eval judges whether completion was honest.

[Michael 'Mike' K. Saleme (msaleme)]

Great question — this is the distinction between agent-reported completion and system-verified completion, and conflating the two is where most multi-agent reliability failures start.

In our testing, the failure modes look like:

1. Downstream agent poisons the result — Worker-B returns "success" along with fabricated data. The orchestrator treats the success signal as proof the task completed correctly, not just that the agent stopped running.
2. Recursive delegation fake-out — A → B → A → B creates a circular chain where each agent reports "the other confirmed," manufacturing the appearance of multi-party consensus without any external anchor.
3. Premature completion claims — an agent reports completion before side effects are actually durable (e.g., "I wrote to the DB" before commit).

A pattern we have found useful is the three-layer proof model:

Layer	What it proves	Who provides it
Agent claim	"I believe I am done"	Agent itself
System verification	"The side effects are durable & match spec"	Orchestrator / sandbox
External confirmation	"An independent witness observed the same result"	Audit / witness agent

For AutoGen, this maps naturally to:

• Agent claim → the agent's reply or terminate message
• System verification → a post-execution check (did the file actually get written? does the DB row exist? does the output schema validate?)
• External confirmation → a second, lightweight "witness" agent that re-executes or re-checks critical steps

The orchestrator should treat agent claims as signals to verify, not as proof of completion. This shifts the trust boundary from "trust the agent" to "trust the verification layer."

We have been testing task-result poisoning and recursive-delegation attacks in an open multi-agent security harness. Happy to share specific test vectors if helpful.

[server (productmakerjason)]

Michael 'Mike' K. Saleme (@msaleme)

This would be very helpful :)

I’m testing a small AoN proof/review record for exactly this boundary: separating agent-reported completion from system-verified completion.

If you’re comfortable sharing one specific test vector, I’d like to convert it into an AoN-style record:

• agent claim
• system verification
• external confirmation, if present
• support level
• accepted scope / failure reason
• receipt candidate or revision state

I’d then share the converted record back here and ask whether it captures the failure mode correctly.

A small synthetic or sanitized example is enough.

[solim (soul-sol)]

The boundary is real, and from having gotten it wrong four times I'd argue it belongs in
neither the framework nor the target application alone — it belongs at the dispatch
site, because that's the only place that knows what proof would be adequate for this
task.

What each layer can and can't do:

• The framework can record that a turn ended and with what terminal status. It cannot
  know that a receipt was required, so it can only ever give you process facts.
• The target application can emit the receipt, but has no idea an agent is waiting on
  it, and usually can't distinguish "not yet" from "never".
• The dispatcher is where the two meet, and where the third state has to exist.

That third state is the whole thing. Two states (done / failed) is what caused every
incident I had:

• COMPLETED — the receiving system returned durable confirmation.
• FAILED — the action provably did not happen. Safe to retry.
• UNVERIFIED — it may have happened; we could not judge. Never auto-retried.
  Re-resolved through an independent channel (query the target for the receipt), then
  collapsed into one of the first two.

Collapsing UNVERIFIED into FAILED is what fires side effects twice. Collapsing it into
COMPLETED is what ships nothing and reports success.

Two traps underneath it, both of which I shipped:

• A clean exit is a candidate, not a verdict. An agent can exit 0 having only asked a
  question — that exact case cost me an afternoon and a second run on top of unfinished
  work.
• The check that decides this can itself fail silently. Mine reported "0 occurrences"
  when the collection had failed, because grep -c exits 1 on a zero count and the
  fallback wrote 0. If your proof-record can't distinguish no receipt from could not
  read receipts, it will eventually record the wrong one.

Reference implementation of the process half, MIT:
https://github.com/soul-sol/agent-watch — exit status recorded out of band, and a missing
completion marker explicitly never means "stalled".

[Adam (KRMAGTU)]

This is the exact assurance boundary I built UAEP to address, but I would push the distinction one step beyond:

completed ⇒ receipt exists

A durable receipt can establish that an operation was accepted or that a particular effect occurred. It does not necessarily establish that the original authorised objective is satisfied in the destination system.

A receipt may be authentic but:

• stale;
• bound to the wrong execution, resource or version;
• evidence of only one part of a multi-effect objective;
• produced before a later conflicting destination change;
• proof of a technical operation that did not satisfy the requested outcome.

UAEP therefore separates:

1. the agent’s completion claim;
2. the executor or tool result;
3. independently observed destination reality; and
4. the assurance verdict.

Its three central checks are:

DestinationConformance — does the observed destination state actually satisfy the authorised objective?

EvidenceApplicability — does the evidence genuinely apply to this objective, execution, resource, version, authority and environment?

RecoveryClosure — after failure, compensation, repair or re-entry, has the original objective actually been closed and independently reverified?

If those facts cannot be established, the truthful result remains UNKNOWN / NOT_VERIFIED rather than COMPLETED or FAILED.

The current UAEP implementation is complete and internally qualified within its declared evaluation boundary. Across the bounded qualified demonstrations:

FALSE VERIFIED = 0

That is not a claim that UAEP cannot fail. The open challenge is to produce a reproducible case where the authorised objective is not legitimately satisfied, yet UAEP returns VERIFIED:

https://community.openai.com/t/i-built-uaep-to-separate-agent-execution-success-from-verified-reality-try-to-make-it-lie/1398398

The useful completion-proof object therefore needs more than evidence that something happened. It needs enough binding and destination evidence to justify the exact objective-level conclusion being claimed.