Zero-dependency multi-agent orchestrator for local, open-weight ML models
Route prompts across local hardware nodes with clean, heavily typed TypeScript. No cloud lock-in. No bloat. No dependencies.
- Why nano-agent-core?
- Features
- Installation
- Quick Start
- Architecture Overview
- API Reference
- Custom Inference Backends
- Examples
- Requirements
- Contributing
- License
The JavaScript ecosystem is flooded with AI packages, but almost all assume you're connecting to centralized, proprietary cloud APIs (OpenAI, Anthropic, etc.). If you want to run inference on your own hardware — Ollama, vLLM, llama.cpp — you're stuck writing a messy web of fetch calls and state management from scratch.
nano-agent-core fixes that.
It gives you:
- Local Node Load Balancing — ping a list of hardware nodes and route prompts to the one with the most available compute.
- Agent State Sync — pass JSON state seamlessly between an extract agent, a reasoning agent, and a formatting agent.
- Zero-Trust Modularity — every tool and agent is an isolated module. Swap models or custom tools mid-pipeline.
- Zero Dependencies — pure TypeScript, uses only Node's built-in
fetchandAbortController. Works in Node 18+ and Edge runtimes.
| Feature | Description |
|---|---|
| 🔄 Load Balancing | 6 strategies: round-robin, weighted, least-latency, least-connections, most-free-memory, least-gpu-util |
| 🧠 Multi-Agent Pipelines | Chain agents with shared state, guards, before/after hooks |
| 🔧 Tool System | Isolated, zero-trust tools with JSON schema validation |
| 🏠 Local-First | Built for Ollama, vLLM, llama.cpp — no cloud required |
| 🪶 Zero Dependencies | Pure TypeScript. Only uses Node built-ins |
| 🖥️ Edge-Ready | Works in Cloudflare Workers, Vercel Edge, Deno (compat) |
| 📦 Tiny Footprint | ~15KB minified. No transitive deps bloat |
| 🔒 Type-Safe | Full TypeScript types out of the box |
bash npm install nano-agent-core
bash yarn add nano-agent-core
bash pnpm add nano-agent-core
typescript import { OllamaProvider, LoadBalancer, createAgent, Pipeline } from 'nano-agent-core';
// 1. Set up the inference provider (Ollama by default) const provider = new OllamaProvider();
// 2. Register your local hardware nodes const lb = new LoadBalancer( [ { id: 'gpu1', url: 'http://localhost:11434' }, { id: 'gpu2', url: 'http://192.168.1.10:11434', weight: 2 }, ], { strategy: 'least-latency' }, provider, );
// 3. Define agents const extractor = createAgent({ id: 'extract', model: 'llama3:8b', systemPrompt: 'Extract key facts from the user input as JSON.', }, provider, lb);
const reasoner = createAgent({ id: 'reason', model: 'llama3:8b', systemPrompt: 'Given extracted facts, reason about them and provide insights.', }, provider, lb);
const formatter = createAgent({ id: 'format', model: 'llama3:8b', systemPrompt: 'Format the reasoning into a clean markdown summary.', }, provider, lb);
// 4. Build a pipeline and run it const pipeline = Pipeline.fromAgents([extractor, reasoner, formatter]);
const result = await pipeline.run( [{ role: 'user', content: 'The sky is blue and water is wet.' }], {}, );
console.log(result.state); // { // extract: { content: '...', servedBy: 'gpu1', ... }, // reason: { content: '...', servedBy: 'gpu2', ... }, // format: { content: '...', servedBy: 'gpu1', ... }, // }
┌─────────────────────────────────────────────────────────────┐ │ Your Application │ └──────────────────────────┬──────────────────────────────────┘ │ ┌──────▼──────┐ │ Pipeline │ ← chains agents with shared state └──────┬──────┘ │ ┌────────────┼────────────┐ │ │ │ ┌─────▼─────┐ ┌────▼────┐ ┌─────▼─────┐ │ Agent 1 │ │ Agent 2 │ │ Agent 3 │ ← each has model, tools, guards │ (extract) │ │(reason) │ │ (format) │ └─────┬─────┘ └────┬────┘ └─────┬─────┘ │ │ │ └────────────┼────────────┘ │ ┌──────▼──────┐ │ LoadBalancer │ ← picks best node per strategy └──────┬──────┘ │ ┌────────────┼────────────┐ │ │ │ ┌─────▼─────┐ ┌────▼────┐ ┌─────▼─────┐ │ Node:GPU1 │ │Node:GPU2│ │ Node:CPU │ ← local hardware nodes │ (Ollama) │ │(vLLM) │ │(llama.cpp)│ └───────────┘ └─────────┘ └───────────┘
Distributes inference requests across a pool of local nodes.
typescript const lb = new LoadBalancer(nodes, options, provider);
Strategies:
| Strategy | Description |
|---|---|
round-robin |
Cycle through nodes sequentially. |
weighted-round-robin |
Nginx-style smooth weighted round-robin (uses node.weight). |
least-latency |
Pick the node with the lowest latency EMA (default). |
least-connections |
Pick the node with the fewest in-flight requests. |
most-free-memory |
Pick the node with the most free VRAM/RAM. |
least-gpu-util |
Pick the node with the lowest GPU utilization. |
Methods:
| Method | Description |
|---|---|
pick() |
Select a node using the configured strategy. |
pickForModel(model) |
Select a node that has the requested model loaded. |
recordStart(node) |
Mark a request as started on a node. |
recordCompletion(node, elapsedMs) |
Record completion and update latency EMA. |
recordError(node) |
Record an error for health tracking. |
healthCheckNow() |
Force an immediate health check on all nodes. |
getHealth() |
Get the latest health snapshot for every node. |
startHealthChecks() / stopHealthChecks() |
Control the background health-check loop. |
Creates an agent that runs inference via the load-balanced provider.
typescript const agent = createAgent({ id: 'extract', model: 'llama3:8b', systemPrompt: 'You are a helpful assistant.', temperature: 0.3, maxTokens: 2048, tools: [searchTool], maxToolIterations: 5, stateSelector: (s) => ({ relevant: s.relevant }), stateReducer: (result, state) => ({ ...state, extract: result.content }), guard: (state) => state.needsExtraction === true, }, provider, lb);
AgentConfig options:
| Option | Type | Description |
|---|---|---|
id |
string |
Unique agent identifier. |
model |
string |
Model name (e.g. llama3:8b). |
systemPrompt |
string |
System prompt for the agent. |
temperature |
number |
Sampling temperature (default: 0.7). |
maxTokens |
number |
Max tokens to generate. |
tools |
Tool[] |
Tools available to the agent. |
maxToolIterations |
number |
Max tool-call rounds (default: 5). |
stateSelector |
(state) => Partial<State> |
Select which state to pass to this agent. |
stateReducer |
(result, state) => State |
Merge agent result into pipeline state. |
guard |
(state) => boolean |
Skip this agent if guard returns false. |
Chains agents together with shared state, guards, and hooks.
typescript const pipeline = new Pipeline([ { agent: extractor, before: (state) => ({ ...state, step: 'extract' }), after: (result, state) => ({ ...state, extractDone: true }), }, { agent: reasoner, // guard: skip if extraction didn't produce results // (guard is on the agent config, not the step) }, ]);
const result = await pipeline.run(initialMessages, initialState);
PipelineResult:
| Field | Type | Description |
|---|---|---|
state |
Record<string, any> |
Final shared state after all agents. |
messages |
Message[] |
Full conversation history. |
stepResults |
StepResult[] |
Results from each step in execution order. |
skippedSteps |
string[] |
Names of steps skipped by their guard. |
elapsedMs |
number |
Total wall-clock time. |
Define an isolated, zero-trust tool that agents can invoke.
typescript import { defineTool } from 'nano-agent-core';
const searchTool = defineTool<{ query: string }>({
name: 'search',
description: 'Search the local document index.',
inputSchema: {
type: 'object',
properties: { query: { type: 'string', description: 'Search query' } },
required: ['query'],
},
execute: async ({ query }, ctx) => {
// ctx.agentId, ctx.state, ctx.nodeId, ctx.signal
return Results for ${query};
},
});
Default inference provider that talks to Ollama's /api/chat endpoint. Also supports OpenAI-compatible mode (apiStyle: 'openai') for vLLM and llama.cpp server.
typescript const provider = new OllamaProvider({ defaultTimeoutMs: 120_000, apiStyle: 'ollama', // or 'openai' });
To use a different backend entirely, implement the InferenceProvider interface:
typescript interface InferenceProvider { complete(node: InferenceNode, request: InferenceRequest, signal?: AbortSignal): Promise; healthCheck(node: InferenceNode, signal?: AbortSignal): Promise; }
nano-agent-core is backend-agnostic. The InferenceProvider interface has two methods: complete and healthCheck. Implement them to support any local inference server — vLLM, llama.cpp, text-generation-webui, LM Studio, etc.
typescript import type { InferenceProvider, InferenceNode, InferenceRequest, InferenceResult, NodeHealth } from 'nano-agent-core';
class VLLMProvider implements InferenceProvider {
async complete(node: InferenceNode, req: InferenceRequest, signal?: AbortSignal): Promise {
const res = await fetch(${node.url}/v1/chat/completions, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
model: req.model,
messages: req.messages,
temperature: req.temperature,
max_tokens: req.maxTokens,
}),
signal,
});
const data = await res.json();
return {
content: data.choices[0].message.content,
model: data.model,
usage: data.usage,
};
}
async healthCheck(node: InferenceNode, signal?: AbortSignal): Promise {
const start = Date.now();
try {
const res = await fetch(${node.url}/health, { signal });
return {
healthy: res.ok,
latencyMs: Date.now() - start,
freeMemory: 0,
gpuUtil: 0,
models: [],
};
} catch {
return { healthy: false, latencyMs: Date.now() - start, freeMemory: 0, gpuUtil: 0, models: [] };
}
}
}
typescript import { OllamaProvider, LoadBalancer } from 'nano-agent-core';
const provider = new OllamaProvider();
const lb = new LoadBalancer( [ { id: 'rtx4090', url: 'http://192.168.1.50:11434', weight: 3 }, { id: 'rtx3090', url: 'http://192.168.1.51:11434', weight: 2 }, { id: 'cpu-fallback', url: 'http://192.168.1.52:11434', weight: 1 }, ], { strategy: 'least-gpu-util', healthCheckIntervalMs: 10_000 }, provider, );
lb.startHealthChecks();
// Pick the best node for the job
const node = lb.pickForModel('llama3:8b');
console.log(Selected: ${node.id} at ${node.url});
typescript import { OllamaProvider, LoadBalancer, createAgent, defineTool } from 'nano-agent-core';
const calculatorTool = defineTool<{ expression: string }>({
name: 'calculate',
description: 'Evaluate a math expression.',
inputSchema: {
type: 'object',
properties: { expression: { type: 'string' } },
required: ['expression'],
},
execute: async ({ expression }) => {
return String(Function(return (${expression}))());
},
});
const provider = new OllamaProvider(); const lb = new LoadBalancer([{ id: 'local', url: 'http://localhost:11434' }], {}, provider);
const agent = createAgent({ id: 'math-agent', model: 'llama3:8b', systemPrompt: 'You are a math assistant. Use the calculate tool for arithmetic.', tools: [calculatorTool], maxToolIterations: 3, }, provider, lb);
const result = await agent.run([{ role: 'user', content: 'What is 17 * 23?' }], {}); console.log(result.content);
typescript import { OllamaProvider, LoadBalancer, createAgent, Pipeline } from 'nano-agent-core';
const provider = new OllamaProvider(); const lb = new LoadBalancer([{ id: 'local', url: 'http://localhost:11434' }], {}, provider);
const classifier = createAgent({ id: 'classify', model: 'llama3:8b', systemPrompt: 'Classify the input as either "technical" or "general".', stateReducer: (r, s) => ({ ...s, category: r.content.trim() }), }, provider, lb);
const techAgent = createAgent({ id: 'tech-responder', model: 'llama3:8b', systemPrompt: 'Respond with technical detail.', guard: (s) => s.category === 'technical', stateReducer: (r, s) => ({ ...s, response: r.content }), }, provider, lb);
const generalAgent = createAgent({ id: 'general-responder', model: 'llama3:8b', systemPrompt: 'Respond in a friendly, simple way.', guard: (s) => s.category === 'general', stateReducer: (r, s) => ({ ...s, response: r.content }), }, provider, lb);
const pipeline = Pipeline.fromAgents([classifier, techAgent, generalAgent]);
const result = await pipeline.run( [{ role: 'user', content: 'How does DNS work?' }], {}, );
console.log(Category: ${result.state.category});
console.log(Response: ${result.state.response});
console.log(Skipped: ${result.skippedSteps});
// Only one of tech-responder or general-responder will run
- Node.js >= 18.0.0 (uses built-in
fetchandAbortController) - ESM only (
"type": "module") - TypeScript 5.4+ (for development)
Contributions are welcome! Please feel free to submit a Pull Request.
- Fork the repository
- Create your feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add some amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
Please make sure to update tests as appropriate and ensure all existing tests pass:
bash npm test
This project is licensed under the MIT License — see the LICENSE file for details.
npm · GitHub · Report a Bug
Made with ⚡ by eyoussef