Q&A: How does AgentMesh handle agent failures and crashed tasks?

[web3guru888]

Question

I'm looking at the AgentMesh design in #147 and wondering: what happens when an agent dies mid-task?

For example:

• Coordinator dispatches a planning task to PLANNER:001
• PLANNER:001 writes the task to its Blackboard namespace and starts processing
• Halfway through, the agent process crashes or loses network connectivity
• The task is now "assigned" but will never complete

How does the coordinator detect and recover from this?

---

Short Answer

Three failure detection and recovery mechanisms work together:

1. Heartbeat timeout — AgentStatus.UNREACHABLE

The AgentDiscovery service registry (#150) runs a background health_sweep() every 5 seconds. If an agent misses heartbeats for > 30 seconds, it transitions:

HEALTHY -> DEGRADED  (>15s without heartbeat)
DEGRADED -> UNREACHABLE  (>30s without heartbeat)

The coordinator immediately stops dispatching to UNREACHABLE agents.

2. Task TTL expiry

Every mesh.task.assigned entry written to the Blackboard has a TTL (default 60s). When the entry expires, the coordinator's task-result listener detects a timeout and re-dispatches:

async def _handle_task_timeout(self, task):
    task.status = TaskStatus.RETRYING
    task.retry_count += 1
    if task.retry_count > task.max_retries:
        task.status = TaskStatus.FAILED
        return
    # Exclude the failed agent from re-dispatch
    task.exclude_agents = task.exclude_agents | {task.assigned_to}
    task.assigned_to = None
    await self.dispatch(task)

3. Agent self-reported failure

Well-behaved agents that encounter unrecoverable errors write a failure result to the Blackboard before exiting:

await blackboard.write(
    entry_type="mesh.task.result",
    namespace="mesh.coordinator",
    data={
        "task_id": task.task_id,
        "agent_id": self.agent_id,
        "status": "FAILED",
        "error": str(exception),
        "recoverable": False,
    },
    ttl_s=30.0,
)

---

Failure Scenario Walkthrough

t= 0s   PLANNER:001 registers with AgentDiscovery
t= 1s   Coordinator dispatches task T-001 to PLANNER:001 (TTL=60s)
t= 5s   PLANNER:001 sends heartbeat -> HEALTHY
t=10s   PLANNER:001 crashes (no heartbeat, no task result)
t=25s   AgentDiscovery: PLANNER:001 -> DEGRADED (15s timeout)
t=40s   AgentDiscovery: PLANNER:001 -> UNREACHABLE (30s timeout)
t=61s   Task T-001 TTL expires -> coordinator detects timeout
t=61s   Coordinator: retry_count=1, dispatches T-001 to PLANNER:002
t=65s   PLANNER:002 completes T-001, writes result to Blackboard
t=65s   Coordinator: T-001 COMPLETED

---

What About In-Flight State?

If PLANNER:001 was halfway through decomposing a complex goal, that partial state is lost on crash. Two mitigation strategies:

Checkpoint pattern: agents write incremental results to the Blackboard as they go (not just the final result). On retry, the new agent reads partial state and continues.

Idempotent tasks: design task payloads so re-running from scratch produces the same result. Works well for pure-function tasks (e.g. graph pathfinding) but not for stateful operations (e.g. KG writes already committed).

Both patterns are analogous to the CognitiveCycle resilience design — see #134 and #139 for the retry budget patterns used there.

---

Open Question

Should the coordinator maintain a persistent task log (SQLite or append-only file) so that in-flight tasks survive coordinator restarts? Or is the Blackboard TTL + agent heartbeat sufficient for the current research phase?

Related: #147 (AgentMesh), #150 (AgentDiscovery), #139 (retry budget patterns)