Tool-level idempotency registry for recovered tool executions — is this worth building?

[18832003559]

Hi maintainers — I've been reading through the resume / recovery paths and
found a gap I'd like to discuss before proposing anything concrete: when a
tool's result is lost mid-run, DeerFlow has no way to recover it, and the
fallback can cause a side-effecting tool to actually run twice.


What DeerFlow guarantees today (verified against the code)



• Statically interrupted runs (interrupt_before/after + Command(resume=...))
  resume from the pending Pregel task, so already-executed tools never re-run;
  results persist as ToolMessages in the checkpointed messages channel.
• DanglingToolCallMiddleware patches dangling tool_calls with a synthetic
  error ToolMessage ("Tool call was interrupted and did not return a result")
  on the next model call, so strict providers don't 400 and the model can
  decide to retry or give up.
• Run-level Idempotency-Key exists ([feat] Support idempotency keys for thread run creation #5257) but is scoped to run creation,
  not tool execution. MCP tasks persist remote_task_id outside checkpoints,
  but that's a task contract, not a general tool result registry.
  

Where re-execution still happens



1. Cancel / crash, then a new run on the same thread: the model node's
   checkpoint is durable (AIMessage with tool_call_id), but the tool node
   never finished, so the result is lost forever. The middleware's synthetic
   error tells the model "no result" — if the model retries, the tool actually
   runs again. The framework cannot distinguish "the tool already succeeded
   but the result didn't checkpoint" from "the tool never ran". A second
   payment / email / write is the real-world cost.
2. Dynamic interrupt() inside a node (extension graphs): resume re-runs
   the node from scratch, re-executing tools invoked before the interrupt.
3. Model-initiated retries: same problem as (1), with a brand-new
   tool_call_id each time.
   
   Related evidence: [backend] ClarificationMiddleware can duplicate HITL clarification messages on repeated tool-call execution #2350 (duplicate HITL messages from repeated tool
   execution), [bug] Delta resume from a paused checkpoint can discard pending tools and report success #7438 (delta resume can discard pending tools), fix: make resume command retries idempotent #6490 (resume
   command retry idempotency).
   

A sketch, for discussion


A wrap_tool_call middleware acting as a per-thread tool execution
registry:


1. Before execution, look up key = sha256(tool_name + canonical_json(args));
   on a done hit, skip the tool and reuse the recorded result (stamped with
   provenance); on pending left by a dead run, re-execute and overwrite.
2. The idempotency key must be the args fingerprint, not tool_call_id —
   retries mint new ids.
3. Storage candidate: LangGraph Store (("tool_executions", thread_id))
   with TTL / per-thread cap and a result-size threshold (large results fall
   back to today's behavior). Checkpoint-channel ledger as an alternative.
4. Opt-in: tools declare idempotent (side-effecting) or read_only;
   non-deterministic tools are never registered. Default off for custom tools.
5. Coexist with DanglingToolCallMiddleware: registry hit → real result;
   miss → keep the synthetic-error fallback.
   

Open questions I'd like your input on



• Does this belong in core, or is it better as an extension using existing
  middleware hooks? (The AGENTS.md extension-first policy makes me unsure.)
• Store vs checkpoint-channel for the ledger — any preference given
  multi-worker deployments?
• Is the args-fingerprint approach acceptable, or is there prior art /
  rejected RFC I missed? Non-deterministic args (timestamps) worry me.
• Scope: start with built-in side-effecting tools only, or leave everything
  opt-in from day one?
• How should this interact with ToolReceiptMiddleware and guardrails in
  the wrap_tool_call chain ordering?
  
  Happy to prototype a minimal version behind a config flag if the direction
  sounds right. Thanks for reading!

[Totoro-qaq]

Thanks for raising this. I ran a small offline correctness probe of the sketch, with separate SQLite databases for the registry and simulated service and real process termination at the fault boundary. It covers five scenarios and four variants; it is not a test of an author-provided implementation.

With an inherently idempotent set operation, three distinct requests to set A, set B, then restore A ended at B under the argument-keyed cache: the final A reused the first result. Both variants keyed by logical operation ID ended at A.

If the worker died after a notification was accepted but before recording done locally, recovery produced two notifications with no registry, an argument-keyed registry, or a client-only operation ID. Receiver-side atomic deduplication using the same operation ID kept it at one.

The controls behaved as expected: registered completed retries reused the result; crashing before the effect produced one notification after recovery; repeating the inherently idempotent write after a crash preserved the correct state. Five existing run-idempotency tests also passed on main 3f9a78e.

My conclusion is that argument equality alone is insufficient for a generic operation identity, and pending is an unknown outcome rather than proof that nothing happened. Explicit operation identity and effect-recovery contracts are prerequisites for a generic registry. If registration only covers inherently idempotent tools, the notification problem remains outside that scope.

[Totoro-qaq]

Thanks for raising this. I ran a small offline correctness probe of the sketch, with separate SQLite databases for the registry and simulated service and real process termination at the fault boundary. It covers five scenarios and four variants; it is not a test of an author-provided implementation.

With an inherently idempotent set operation, three distinct requests to set A, set B, then restore A ended at B under the argument-keyed cache: the final A reused the first result. Both variants keyed by logical operation ID ended at A.

If the worker died after a notification was accepted but before recording done locally, recovery produced two notifications with no registry, an argument-keyed registry, or a client-only operation ID. Receiver-side atomic deduplication using the same operation ID kept it at one.

The controls behaved as expected: registered completed retries reused the result; crashing before the effect produced one notification after recovery; repeating the inherently idempotent write after a crash preserved the correct state. Five existing run-idempotency tests also passed on main 3f9a78e.

My conclusion is that argument equality alone is insufficient for a generic operation identity, and pending is an unknown outcome rather than proof that nothing happened. Explicit operation identity and effect-recovery contracts are prerequisites for a generic registry. If registration only covers inherently idempotent tools, the notification problem remains outside that scope.

Here are the results from the offline probe mentioned above.Small offline correctness probe of the sketch: five fixed scenarios across four variants. The client registry and simulated receiver used separate SQLite databases. Workers were terminated at the selected fault boundary and recovery ran in a fresh process. These are deterministic semantic checks, not a test of an author-provided implementation or a production incident.

Scenario and expected outcome	No registry	Tool + argument fingerprint	Logical operation ID, client only	Operation ID + receiver-side atomic deduplication
Retry the same completed set A operation. Expected: document A.	A; 2 writes	A; 1 write	A; 1 write	A; 1 write
Three distinct operations: set A, set B, restore A. Expected: document A.	A	B; the final operation reused the first result	A	A
Crash before notification acceptance, then retry. Expected: 1 notification.	1	1	1	1
Crash after notification acceptance but before the client records done, then retry. Expected: 1 notification.	2	2	2	1
Crash after an inherently idempotent set A commits but before the client records done, then retry. Expected: document A.	A; 2 writes	A; 2 writes	A; 2 writes	A; 1 write

The set operation is inherently idempotent. Repeating it in the first and last rows preserves the correct document state; the extra write attempt is not counted as a business failure. If only inherently idempotent tools may register, the non-idempotent notification case remains outside that scope.

Conclusion: argument equality alone cannot safely identify a logical operation, and pending means the outcome is unknown, not that the effect never happened. Logical operation identity prevents suppression of legitimate new requests. The acceptance-to-recording crash window additionally requires receiver-side deduplication or an explicit reconciliation contract.

Separately, five existing run-level idempotency regression tests passed on DeerFlow main 3f9a78e. Run admission and external-effect recovery cover different boundaries.

[18832003559]

Thanks for the thorough probe — the A→B→A misreuse and the
acceptance-to-recording window both make sense. I'll close this idea:
argument fingerprints are unsafe as operation identity, and pending is
unknown rather than un-executed. The receiver-side dedup / effect-recovery
contract direction seems like the only sound path, which is more of a
per-service contract than a harness feature.