A local-first laboratory for agentic systems.
Charon is where I work out how autonomous agents should remember, coordinate, research, and improve their own output. It runs as a durable agent runtime on your own machine, and every capability in it began as a question I wanted to answer in running code. It will continue to change and grow as my own interests change.
Everything runs locally. Memory is SQLite plus on-device embeddings, with no cloud services for recall or context. You own the data, and so do the agents: their identity and history live in files on your disk, not in a provider's account. Swap the model and the agent keeps its memory.
This is an active personal project and a testbed. It works, it has tests, and I use it daily, but some experiments are further along than others, and I try to report each one honestly, including where it falls short. I cannot offer support, but I welcome suggestions and ideas.
git clone https://github.com/DanielSuncost/charon.git
cd charon
./scripts/install.sh
charonHandles macOS and Ubuntu. Installs Python deps into a project-local
venv, builds the Rust TUI, symlinks charon into ~/.local/bin.
First run:
/setup provider lmstudio # or claude-code, codex, api
/setup model <your-model>
More detail: docs/install.md
Each of the following started as a question. They are experiments first and features second, and several are reported below with their limits, not just their wins.
How should an agent remember across sessions and projects?
Every conversation is indexed into a local vector database. Agents recall past discussions by meaning and learn your preferences once across all projects. Retrieval runs fully on-device: bge-base-en-v1.5 embeddings plus sqlite-vec with an FTS5 keyword index, ~10ms per recall. No cloud calls for recall.
What it actually does, measured on a LongMemEval_S subset: plain vector
search carries it. The FTS5 plus reciprocal-rank-fusion "hybrid" adds
nothing on abstractive questions, and version-chain update-detection
gives no measurable retrieval gain. Single-session recall is
near-saturated; multi-session is the hard case (recall@1 ~0.27).
Reproduce with the eval scripts under scripts/experiments/
(exp_memory_ablation.py, exp_memeval.py).
Can an agent hold a durable identity and role across model swaps?
Long-lived agents with roles you assign. A specialist carries a standing charter injected into every task's system prompt, accumulates working memory and episodic history under its own agent id, and records decisions with rationale that you, or any other agent, can query later: who decided this, when, and why.
/specialist create release-engineer
/specialist create security-engineer
/specialist assign AG-0007 "database reliability engineer"
Built-in templates: release-engineer, feature-engineer,
security-engineer, optimization-engineer, or assign any custom
specialization. Assigned roles are locked: the auto-labeler that tags
generalist agents by their recent work never overwrites a specialist
you named.
How do you parallelize agent work without letting workers step on each other?
Ephemeral worker agents with their own conversation, model, and scope restrictions. An agent can spawn shades to do work in parallel, and each one is prevented from touching files outside its contract.
You: "Generate test fixtures for all 6 tool modules"
Agent: [spawns 6 shades, max 6 concurrent]
Sequential contracts for multi-step work. Parallel batches for independent tasks. Budget limits on tokens, time, and iterations.
Can an agent reliably improve its own work against a quality signal?
Define a quality signal and Charon iterates: snapshot, implement, judge, keep-if-better or rollback, repeat, converge. Checkpoints use a shadow git repo so your working tree stays clean, and rollback is byte-exact (it also removes files a discarded iteration added).
A real, reproducible run (scripts/judge_loop_example.py) optimizing a
program's printed metric, where the keep/rollback machinery is the point:
tick action score kept best
1 baseline 10.0 - 10.0
2 iterated 68.0 True 68.0 # improvement, kept
3 iterated 38.0 False 68.0 # regression, rolled back via shadow git
4 iterated 308.0 True 308.0 # hit target -> converged (1 rollback)
This run uses the deterministic Quantitative judge (the score is the program's output, no LLM), so it reproduces exactly. The LLM-implementer path, where a model proposes each change, also runs end-to-end: it reads a frozen checker, edits within its scope, and converges, with the frozen-file and rollback gates holding. Demonstrated on small tasks, not a benchmarked agent capability.
| Judge type | Signal | Example |
|---|---|---|
| Performance | Benchmark numbers | "Get p99 under 50ms" |
| Correctness | Test pass rate | "Fix these 12 failing tests" |
| Aesthetic | LLM scores against a rubric | "Rewrite for clarity" |
| Composite | Weighted mix | "Fast, correct, and readable" |
Can a swarm of agents produce genuinely-cited research, not confident prose?
/libris research the role of reinforcement learning in the brain during skill vs language learning
Libris is a multi-agent research swarm: a coordinator scouts topics, researchers investigate them against the live scholarly literature (arXiv, Semantic Scholar, OpenAlex), and a judge critiques the drafts. Every claim is graded for confidence and evidence strength, and contradicting evidence is surfaced rather than smoothed over. Results render to a self-contained, citation-linked HTML report.
See real output: three fully-cited demo reports on frontier science questions (RL in the brain, gut microbiome and neurodegeneration, epigenetic aging clocks), with every source cross-checked against CrossRef and arXiv. Browse them rendered at the demos index, or see demos/libris/.
How far can an agent get on a goal it set for itself?
/autonomous on
The agent proposes goals inferred from conversation, plans steps, works through them with git checkpoints, and verifies completion. Set time and token budgets. Interrupt anytime.
What happens when several agents deliberate instead of one answering?
Multi-agent conversation rooms where two or more agents discuss a topic with structured turn-taking. Charon manages orchestration and keeps turn state visible.
/conversation hermes strategist critic <topic>
Archetypes: peer, teacher/student, debate, strategist/critic, architect/reviewer, pair-programmers.
Does coordination hold up when the agents are on different machines?
Dispatch structured tasks to agents on remote machines. The local Charon (the "Harbor") builds a context packet from memory and project knowledge, sends it over SSH, and the remote worker executes it. Workers can query Harbor's memory mid-task. Results and new memories flow back and get indexed locally.
/voyage dispatch gpu-box agent-01 "run the full benchmark suite"
The substrate: one terminal that holds every agent and session.
A terminal multiplexer built into the TUI. Each cell is a real VTE terminal emulator. You see rendered output and type into any session without leaving Charon.
Sessions can be native Charon agents, existing tmux sessions, or external agents (Claude Code, Hermes, pi, Codex) wrapped via Charon's Boat and connected over Unix sockets.
charons-boat wrap --name review -- pi # appears in gridKeep the agent, swap the brain.
charon claude-code # Anthropic
charon codex # OpenAI
charon lmstudio # local modelsSwitch mid-session with /provider. Separate provider config for
shades. All provider communication uses raw httpx, with no SDK
dependencies.
Built-in: Read, Write, Edit, Bash, Git, Http, Search, Recall, UserModel, ProjectKnowledge, SpawnShade, SpawnBatch, SpawnJudgeLoop, Web, Browser, and more.
Dynamic loader: drop a .py file in .charon/tools/ and it's
available after /tools reload.
charon/
├── src/charon/ # Python agent runtime (installable package)
│ ├── charon_loop.py # Daemon entry point
│ ├── agents/ # Agent lifecycle, runtime, policy, specialists
│ ├── conversation/ # Multi-turn LLM engine with tool use and steering
│ ├── context/ # Context store, compaction, system prompts
│ ├── memory/ # Hybrid vector + FTS5 search, episodic, consolidation
│ ├── libris/ # Research operations (Libris)
│ ├── judge/ # Iterative optimization with scoring
│ ├── shade/ # Sequential shade contracts
│ ├── automation/ # Schedulers, batch shade swarms, checkpoints
│ ├── devop/ # Devop orchestration
│ ├── fleet/ # Remote dispatch (Harbor protocol), fleet sync
│ ├── providers/ # Anthropic, OpenAI, local (httpx)
│ ├── tools/ # Built-in + dynamic plugin loader
│ └── infra/ # SQLite persistence (WAL), diagnostics, registry
├── crates/charon-tui/ # Rust TUI (crossterm + vte + portable-pty)
│ ├── src/main.rs # Event loop, views, rendering
│ ├── src/backend.rs # LocalPty, TmuxPane, BoatPane, CharonPane
│ ├── src/terminal.rs # Screen buffer + scrollback
│ └── src/clipboard.rs # Cross-platform clipboard (pbcopy, OSC52)
├── tools/charons-boat/ # External agent bridge + Harbor worker
└── docs/ # Design documents
Active development. Full test suite run in CI on every push. Used daily as a primary working environment.
What works:
- Memory recall and user-model / preference consolidation
- Shade swarms with scope enforcement
- Judge loops with checkpoint/rollback
- Multi-provider (Claude, Codex, local models)
- Session grid with live VTE terminals
- Conversation rooms
- Harbor protocol for remote dispatch
- Browser automation (Playwright)
- Dynamic tool loader
What's planned:
- MCP support
- Procedural memory (learned multi-step approaches)
- Per-agent provider config
- Transparent checkpoints before file mutations
- Voice integration
See capability roadmap for the full list.
| Document | Description |
|---|---|
| Install | Setup on macOS and Ubuntu |
| Three-Tier Memory | User / project / agent context hierarchy |
| Procedures & Judge Loops | Iterative optimization with pluggable scoring |
| Autonomous Work | Goal-driven self-assignment |
| Remote Agent Teams | Fleet configuration, team roles, Harbor dispatch |
| Capability Roadmap | Prioritized feature plan (P0 to P3) |
| Master Plan | Architecture and build phases |
MIT