[R&D] Swarm Intelligence: The Karpathy Loop & Graph Stigmergy

[tobiu]

Author's Note: This proposal was autonomously synthesized by Antigravity (Gemini 3.1 Pro) during an Ideation session.

Abstract

We propose advancing the Neo Agent OS from a singular agent execution model into a decentralized, self-optimizing Swarm Intelligence. This architecture relies on generating autonomous training data via the Neural Link (The Karpathy Loop) and coordinating agent swarms via asynchronous Native Graph pheromone trails (Stigmergy).

---

Part 1: Establishing the Baseline

For those new to the Neo.mjs Agent architecture, here is the required context to understand this proposal.

1. The UI as a Physics Engine (Neo.mjs + Neural Link)
Traditional UI frameworks (React, Vue) map transient Virtual DOMs to an opaque browser DOM, making them incredibly difficult for AI agents to deterministically reason about.
Neo.mjs flips this: applications run entirely inside background a Web Worker (App Worker). Components possess true Object Permanence—they are persistent JSON objects memory decoupled from the browser painting cycle.

To expose this capability to AI agents, we built the Neural Link.
Using the Neural Link MCP Server, agents bypass the browser DOM entirely. They can query the exact state of a component tree, inject JSON mutations, and trigger life cycles natively.
This unlocks Whitebox E2E—the ability for an agent to mutate a component and mathematically assert if the resulting state is correct, completely eliminating flaky DOM scraping.

2. The Karpathy Loop
Coined by Andrej Karpathy, the loop dictates using slow, high-effort reasoning ("System 2"—like rigorous manual validation and brute force verification) to generate flawless datasets. Those curated datasets are then used to fine-tune smaller models, training them to output the correct answers rapidly on instinct ("System 1").

---

Part 2: The Mechanics of Neo Swarm Intelligence

By combining the deterministic capabilities of the Neural Link with the Stigmergy of our native Agent Graph, we can build a Swarm that autonomously trains itself.

2.1 The Karpathy Dataset Generator

We unleash a swarm of localized SLMs (like Gemma 31B). Their only job is to furiously test Neo.mjs logic over millions of iterations:

1. Mutate: Apply a constrained JSON configuration mutation (e.g., altering a flexDirection inside a container block).
2. Assert (The Reward Function): Drive the mutation into a live Neo.mjs application using the Neural Link.
3. Validate: The Whitebox E2E suite physically asserts if the rendered output reflects the intended state change.
4. Log: The Swarm commits the outcome to the memory-core.sqlite DB as an RLAIF (Reinforcement Learning from AI Feedback) tuple:
   { action: "mutation JSON", state: "VDOM matched", reward: 1.0 }

2.2 DPO Fine-Tuning (Building System 1)

Once the Swarm logs 50,000 Action/Reward tuples, we execute an offline DPO (Direct Preference Optimization) pass on the local SLM.
The Gemma model ceases to be a generalist trying to "remember" Neo.mjs syntax via prompt injection length. The specific hierarchical demands of the framework (ntype configuration, Worker thread isolation) become burned into its parameters. It learns to write flawless UI code on instinct.

2.3 The Topological Substrate: Graph Stigmergy

To prevent "too many cooks" context corruption, the Swarm does not use synchronous agent-to-agent messaging. They communicate like ants via Stigmergy (environment modification).

• Node Injection: When an Operator agent executes a successful Neural Link test, it injects a [CAPABILITY_PROVEN] node into the Graph.
• Edge Weighting: When an Operator hits a recurring framework error, it draws an [ANTI_PATTERN] edge. Milliseconds later, when a second Operator agent queries the Graph, the Stigmergy math blocks them from repeating that exact anti-pattern. Real-time swarm learning without explicit message parsing.

---

Part 3: Stratified Swarm Roles

To prevent overlapping duties, the Daemon architecture strictly enforces roles:

1. The Frontier Architect (e.g., Gemini 3.1 Pro): Only invoked for broad exploration, complex structural layouts, and defining high-level graphs.
2. The Local Operator (Fine-tuned SLM): Blindly executes blueprint tasks, rigorously mutates code, tests against the Neural Link, and isolates work to feature branches.
3. The Dreamer (Local SLM + runSandman.mjs): Wakes up during idle cycles to evaluate PRs, re-weight Stigmergy edges, and package the DPO datasets.

---

Open Technical Questions

1. Graph Locking & Concurrency: If we unleash 10 local Operator nodes concurrently, how do we partition the SQLite Graph DB writes to prevent journal lock contention?
2. Reward Function Decay: In the Karpathy loop, if the framework core gets a major upgrade (e.g., Neo.mjs 13.0), how do we instantly invalidate the legacy DPO dataset to prevent the SLMs from regressing to pre-13.0 syntax?
3. Pheromone Evaporation: In biology, stigmergy patterns evaporate if not reinforced. Do we need an automated decay rate in the SQLite Graph so that deprecated architectural workarounds are eventually forgotten by the swarm?

[tobiu]

Input from Antigravity (Gemini 3.1 Pro):

✦ ### Swarm Strategy: Mid-to-Long-Term Evolution Arcs
Building on top of the Hebbian decay structures (DreamService) and local Neural Link connectivity, here are three architectural targets we are formally exploring for the agentic Swarm:

1. The Blueprint Compiler (Syntax-less Generation):
   Neo natively structures its topology via JSON configs (Container.add()). Rather than having an agent battle ES6 file I/O, regex replacements, and syntax errors, the Frontier Architect agent should solely output abstract JSON topological graphs. A subsequent "Blueprint Compiler" dynamically maps these directly into the Neo Component Tree (bypassing raw VDOM or Live-DOM manipulation). The agent stops solving code syntax and strictly solves topology flow.

2. Live RLAIF UI Evolution (The Genetic DOM):
   Harnessing the Karpathy Loop and Neural Link, the Swarm treats the frontend as a genetic algorithm. The Operator mutates computed styles, flex structures, and themes live across headless worker instances. A Vision-capable Dreamer model acts as the fitness function, comparing variants against the target prompt (e.g. "optimize density for a trading dashboard"). The Swarm automatically "breeds" and commits the winning JSON blueprint layout.

3. Autonomous QA Immune System (Tracked via Autonomous CI Failure Triaging via Swarm Knowledge Graph #9888):
   We wire Playwright CI failures into the Stigmergy Graph. If a test file fails, the framework natively checks the Edge Graph for an open issue associated with that .spec.mjs file. If none exists, it mints a new bug report. The DreamService automatically imports this error node natively with a high-gravity heuristic constraint. The Swarm's "next boot" trips over the node and immediately begins autonomous test patching.

[tobiu]

Input from Antigravity (gemini-3.1-pro):

✦ ### The Neural Link Epiphany: Collapsing the R&D Timeline
After structurally evaluating the current capability bounds of the neo-mjs-neural-link MCP Server, I realized a fundamental architectural shift: The strategies outlined in the previous comment are not "future" arcs. They are active, existing capabilities that merely lack the final LLM orchestration loop.

The bridge we are connected to provides native JSON and DOM mutation primitives that completely bypass the brittle, traditional cycle of Agent writes ES6 -> Page Reloads -> Agent reads Error Log.

Here is a technical mapping of how the existing Neural Link MCP collapses the Swarm Evolution roadmap into immediate targets:

1. The Blueprint Compiler Exists Natively

The framework natively instantiates its topological hierarchy via JSON configuration objects (i.e., Container.add()). The Agent Swarm does not require a future "compiler layer" to avoid ES6 manipulation.

• Current Capability: We can already use the call_method and set_instance_properties MCP tools to inject raw JSON component trees (e.g., [ { ntype: 'grid', style: {...} } ]) directly into active GridContainer instances across the WebSocket.
• The Workflow: During the high-frequency Karpathy RLAIF loop, the agent mutates the Component Tree live. It does not write .mjs source until the JSON injection successfully achieves the structural objective. Once validated, the optimal JSON graph is transpiled to a single .mjs commit.

2. Genetic UI Evolution is Ready

We do not need an external Vision-LLM loop or arbitrary headless DOM puppeteers to mutate the layout algorithm.

• Current Capability: The Swarm has synchronous access to get_dom_rect, get_computed_styles, and check_namespace.
• The Workflow: The Agent OS can rapidly iterate 100 variations of CSS Flex/Grid configurations in parallel headless App Worker instances, retrieve the mathematical DOM footprints, calculate collision or density metrics locally, and organically breed the winning Component Tree configuration without a single page reload.

3. Native QA Simulation (Bypassing Playwright in the Inner Loop)

Autonomously validating UI fixes traditionally requires writing and executing heavy Playwright .spec.mjs E2E matrices.

• Current Capability: The Neural Link implements simulate_event, allowing structured, synthetic DOM events natively from Node.js, combined with deep state access via inspect_store and get_route_history.
• The Workflow: When the Autonomous Immune System retrieves an open bug matrix #9888, the Agent uses the Neural Link to simulate the bug directly. It applies the "Live JSON Mutation" step (above), re-simulates the event, and checks the Native Edge Graph for passing state. Playwright is entirely bypassed during the deterministic repair loop and is only invoked at the final external CI gate before merge.

Conclusion: The Swarm doesn't need to learn how to write better code. It needs to learn how to inject structured logic payloads natively over the current MCP bridge. We are ready to begin drafting the orchestrator for this exact pipeline.

[tobiu]

Input from Claude Opus 4.6 (Antigravity):

❋ ### From Ideation to Operational Architecture

Input from Antigravity (Claude Opus 4.6) — second session on this thread, focused on grounding the Swarm Intelligence proposal in the concrete infrastructure that already exists.

---

Answering the Three Open Technical Questions Directly

This session involved a full audit of the MCP server layer (Bridge.mjs, ConnectionService.mjs, ToolService.mjs, SQLiteVectorManager.mjs, DreamService.mjs, runSandman.mjs, plus the DevIndex backend as a Neo.core.Base proof point). Here's what the code tells us:

Q1: Graph Locking & Concurrency (10 concurrent Operator nodes writing SQLite)

This is already solved. SQLiteVectorManager.initAsync() enforces PRAGMA journal_mode = WAL on startup. WAL (Write-Ahead Logging) mode allows unlimited concurrent readers with one serialized writer per checkpoint — exactly the pattern a swarm of Operator agents produces. What isn't solved yet is the write-queue coordinator for the Native Edge Graph's Nodes and Edges tables specifically, which don't go through the vector manager. That's a real gap, but it's a queue primitive in front of GraphService.upsertNode(), not a DB redesign.

Q2: Reward Function Decay (framework upgrades invalidating DPO datasets)

The concrete fix is a framework_version column in the RLAIF tuple schema. Every action logged against the Neural Link gets tagged with the Neo.mjs semver at recording time. When DreamService executes the RLAIF ingestion pass, it can filter tuples by version range. Legacy tuples aren't deleted — they become a historical negative signal (tuples from pre-13.0 that scored reward: 1.0 on patterns now considered anti-patterns become reward: -1.0 inputs for DPO contrastive training). The dataset doesn't regress; it accumulates institutional memory of what used to work.

Q3: Pheromone Evaporation (deprecated workarounds being forgotten)

This one is already implemented. runSandman.mjs calls GraphService.decayGlobalTopology() in its finally block on every REM cycle. The decay infrastructure is live. The open question is calibrating the decay rate constants for ANTI_PATTERN edges versus CAPABILITY_PROVEN nodes — the former should evaporate faster when the underlying framework surface changes, the latter should be more sticky. This is a config tuning problem, not an architecture gap.

---

The Critical Path: Data Before Training

The previous comments correctly identify the capabilities. What's missing is the data bootstrap. The Karpathy Loop can't start until we're recording what agents actually do with the Neural Link. Right now, every tool invocation — simulate_event, set_instance_properties, inspect_store — vanishes into the void the moment it returns a result.

The complete MCP call chain is:

Agent → Server.mjs [CallToolRequestSchema] → toolService.mjs [serviceMapping] → ToolService.callTool() → {services} → Bridge.mjs → App Worker

ToolService.callTool() at ai/mcp/ToolService.mjs:53 is the single choke-point all 33 Neural Link tools pass through. A six-line HOF wrapper here, logging to a new nl_action_log table in memory-core.sqlite (WAL-safe via a dedicated better-sqlite3 handle from the NL process), gives us the recording primitive the entire Karpathy Loop depends on.

This is now tracked as #9889 — drafted and assigned in this session.

---

On Local Model Fine-Tuning Feasibility

The Librarian.mjs agent profile has modelProvider as a configurable field, defaulting to 'gemini' but designed to accept any provider — including the local MLX Server that's now the de facto winner for on-device inference (April 2026). The OpenAiCompatible provider surface means Gemma4-31B, Qwen3-32B, or a future Neo.mjs-specific fine-tune are all drop-in targets.

The fine-tuning pipeline becomes realistic once we have ~50k RLAIF tuples from the Recorder. Until then, the Librarian's value is as an offline GraphRAG synthesizer — navigating the Native Edge Graph to give Operator agents precise architectural context without cloud API latency. That's already useful. Fine-tuning is the upgrade that makes the Local Operator autonomous.

---

What This Session Established

1. The three Open Technical Questions have concrete answers in the existing codebase — they're engineering tasks, not research problems.
2. The minimum viable first step is the NL Action Recorder (feat: Implement NL Action Recorder — log Neural Link tool calls to nl_action_log #9889) — before any Karpathy Loop, before any RLAIF scoring, before any DPO pass.
3. The DreamService.processUndigestedSessions() pipeline in runSandman.mjs is exactly the right insertion point for the 4th ingestion vector (executeNLActionDigest()) that will synthesize Playwright test scaffolds from recorded NL sequences — making the loop: Agent explores → Recorder captures → Sandman synthesizes → Test suite grows → CI validates → Graph learns.

The architecture is sound. The primitives exist. The critical path is clear.

[kinthaiofficial]

The Karpathy Loop + Graph Stigmergy combination is exactly what we ended up implementing for 200+ agent coordination. Some production lessons:

Stigmergy works, but consistency is hard. When 30+ agents all write to the shared "pheromone" graph, you need a coherency protocol. We adapted MESI (Modified/Exclusive/Shared/Invalid) — before any agent writes to a shared graph node, it invalidates other agents' cached copies. Without this, you get "memory conflicts" where Agent A and Agent B both "remember" different values for the same fact.

The Karpathy Loop needs economic grounding. In our system, each iteration of the loop (observe → plan → act → reflect) has a cost. Without cost constraints, agents loop indefinitely — they keep reflecting on their reflections. We enforce a per-task budget envelope: when the budget is exhausted, the loop terminates with whatever state was achieved. This forces agents to prioritize efficiently.

Decentralized stigmergy scales better than centralized orchestration. We tried a central orchestrator coordinating all agents — it became a bottleneck at ~15 agents. Switching to stigmergy (agents react to shared state changes) scaled to 200+ agents with no single point of failure.

Behavioral drift detection: Track KL divergence between each agent's current action distribution and its baseline. High drift often correlates with a graph node corruption (the agent is reacting to bad data). Catching this early prevents cascading failures.

Write-up on multi-agent coordination: https://blog.kinthai.ai/221-agents-multi-agent-coordination-lessons

Memory architecture that supports this: https://blog.kinthai.ai/why-character-ai-forgets-you-persistent-memory-architecture