Skip to content

fix(agent): back off failed VM reconcile sweeps - #1682

Merged
branarakic merged 5 commits into
mainfrom
fix/backoff-vm-reconcile-failures
Jul 14, 2026
Merged

fix(agent): back off failed VM reconcile sweeps#1682
branarakic merged 5 commits into
mainfrom
fix/backoff-vm-reconcile-failures

Conversation

@branarakic

Copy link
Copy Markdown
Contributor

Summary

  • let failed VM reconcile passes reach the per-CG coalescer after they are logged
  • suppress queued/live-nudge retries for one configured sweep interval after a failure
  • retain immediate coalescing and trailing runs for successful reconciliation
  • add deterministic coverage for failure cooldown and retry at the boundary

Mainnet evidence

10.0.6 cores repeatedly report 30-second Blazegraph deadlines while reconciling the same on-chain graphs (foodie-network on Gnosis and sports on Base). Several nodes produced roughly 15 VM timeout warnings in a 15-minute sample.

runVmReconcileForCg currently catches the timeout and resolves normally. The coalescer therefore cannot distinguish failure from success: a KACG event queued during the expensive scan immediately launches the trailing scan, and later live events can keep retriggering it despite the periodic safety net.

After this change, a failed CG is cooled down for DKG_VM_RECONCILE_INTERVAL_MS (10 minutes on the inspected mainnet units). The periodic sweep retries when that window expires. Successful live-event reconciliation is unchanged and remains immediate.

This is orchestration-only: it does not delete subscriptions or graph data and does not advance a failed reconciliation cursor.

Verification

  • pnpm --filter @origintrail-official/dkg-agent exec vitest run --config vitest.unit.config.ts test/chain-reconciler.test.ts
  • pnpm --filter @origintrail-official/dkg-agent exec tsc --noEmit
  • pnpm --filter @origintrail-official/dkg-agent test:unit (52 files, 593 tests)

Comment thread packages/agent/src/dkg-agent-swm-host.ts Outdated
Comment thread packages/agent/src/chain-reconciler.ts Outdated
Comment thread packages/agent/test/chain-reconciler.test.ts
@branarakic
branarakic merged commit 4c5b108 into main Jul 14, 2026
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants