Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 94 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ AI coding agents waste tokens exploring codebases. Every grep, glob, and file re
| **70+ MCP Tools** | **50+ Languages** | **12+ Agent Integrations** |
| From call graph traversal to dead code detection, atomic edit primitives, code-health metrics, test mapping, and complexity analysis. | Rust, Go, Java, Python, TypeScript, C, C++, Swift, Svelte, Astro, and 42 more including WGSL/HLSL/Metal shaders and Markdown. Three tiers (lite/medium/full) control binary size. | Claude Code, Codex CLI, Gemini CLI, Hermes, Kiro, Cursor, OpenCode, Copilot, Cline, Roo Code, Zed, Antigravity, Kilo CLI, Kimi CLI, Mistral Vibe. |
| **Multi-Branch Indexing (opt-in)** | **100% Local** | **Always Fresh** |
| Optional per-branch databases. Cross-branch diff and search without switching your checkout. | No data leaves your machine. No API keys. No external services. Everything runs on a local libSQL database. | On-demand staleness check on every MCP call (30 s cooldown) plus catch-up sync when the server connects. Multi-agent work is expected to use git worktrees — each agent gets its own checkout and the index diverges are merged by git, not by a file watcher. |
| Optional per-branch databases. Cross-branch diff and search without switching your checkout. | Source code and memory content stay on your machine. No API keys or hosted database are required; the index runs on local libSQL. | On-demand staleness check on every MCP call (30 s cooldown) plus catch-up sync when the server connects. Multi-agent work is expected to use git worktrees — each agent gets its own checkout and the index diverges are merged by git, not by a file watcher. |
| **Subprocess-Isolated Extraction** | **Code-Health Analytics** | **Atomic Edit Primitives** |
| A native crash in any tree-sitter grammar (abort, segfault, anything) kills only the worker; the pool respawns it and sync continues. Sync never dies on a malformed file. | Composite health score (0-10000), Gini inequality, file-DAG depth, design-structure matrix, risk-weighted test gaps, and session deltas. | Edit files without regex or shell-quoting hazards: unique-anchor `str_replace`, atomic multi-replace, AST-rewrite, anchored insert. Auto re-indexes after writes. |

Expand Down Expand Up @@ -147,11 +147,11 @@ For project-scoped setup, run from the repository root:
tokensave install --local --agent cursor
```

Local install writes only workspace files such as `.cursor/mcp.json`, `.mcp.json`, `.codex/config.toml`, `.vscode/mcp.json`, `.hermes/plugins/tokensave/`, or the equivalent project config for Claude, Codex, Gemini, Hermes, Kiro, OpenCode, Copilot/VS Code, Zed, Roo Code, Kimi, Kilo, and Vibe. Generated MCP configs and plugin wrappers use the resolved absolute `tokensave` executable path. Hermes installs into `~/.hermes/plugins/tokensave/` by default, or into `~/.hermes/profiles/<name>/plugins/tokensave/` with `--profile <name>`; profile names are normalized to lowercase and must match `[a-z0-9][a-z0-9_-]{0,63}`. Use `tokensave uninstall --agent hermes --profile <name>` to remove a named profile install; `reinstall` and `doctor --agent hermes` currently operate on the default profile. Hermes wrappers run from Hermes' current working directory, use a 600-second timeout, and include truncated stdout/stderr in error JSON. Hermes local install without `--profile` writes only project plugin files and `.hermes/config.yaml`; `tokensave install --local --agent hermes --profile <name>` is a deliberate mixed-scope mode that targets the named profile instead. Hermes requires `HERMES_ENABLE_PROJECT_PLUGINS=true` when launching with project-local plugins. For Cursor, local install also writes `.cursor/rules/tokensave.mdc`, `.cursor/permissions.json`, and `.cursor/hooks.json`: the rule tells Cursor Agent to prefer tokensave MCP tools for codebase exploration, and permissions auto-allow only read-only tokensave MCP tools. The project hooks are:
Local install writes only workspace files such as `.cursor/mcp.json`, `.mcp.json`, `.codex/config.toml`, `.vscode/mcp.json`, `.hermes/plugins/tokensave/`, or the equivalent project config for Claude, Codex, Gemini, Hermes, Kiro, OpenCode, Copilot/VS Code, Zed, Roo Code, Kimi, Kilo, and Vibe. Generated MCP configs and plugin wrappers use the resolved absolute `tokensave` executable path. Hermes installs into `~/.hermes/plugins/tokensave/` by default, or into `~/.hermes/profiles/<name>/plugins/tokensave/` with `--profile <name>`; profile names are normalized to lowercase and must match `[a-z0-9][a-z0-9_-]{0,63}`. Use `tokensave uninstall --agent hermes --profile <name>` to remove a named profile install; `reinstall` and `doctor --agent hermes` currently operate on the default profile. Hermes wrappers run from Hermes' current working directory, use a 600-second timeout, and include truncated stdout/stderr in error JSON. Hermes local install without `--profile` writes only project plugin files and `.hermes/config.yaml`; `tokensave install --local --agent hermes --profile <name>` is a deliberate mixed-scope mode that targets the named profile instead. Hermes requires `HERMES_ENABLE_PROJECT_PLUGINS=true` when launching with project-local plugins. For Cursor, local install also writes `.cursor/rules/tokensave.mdc`, `.cursor/permissions.json`, and `.cursor/hooks.json`: the rule tells Cursor Agent to prefer tokensave MCP tools for codebase exploration, and permissions auto-allow the full local tokensave MCP surface for that workspace. The project hooks are:

- `sessionStart` — fire-and-forget; injects context steering the Agent toward tokensave MCP tools and reports index freshness (suggests `tokensave init` when no `.tokensave/` exists).
- `subagentStart` — blocks research/explore subagents until tokensave MCP tools have been tried.
- `beforeSubmitPrompt` — resets the local token counter for the new turn.
- `beforeSubmitPrompt` — resets the local token counter for the new turn and ingests the current Cursor transcript into `.tokensave/sessions.db` when `transcript_path` is present.
- `afterFileEdit` (matcher `Write`) — runs a **targeted single-file** sync of just the edited path(s) via `sync_if_stale_silent`, never a full-tree scan (which would scale with repo size, not edit size).
- `afterShellExecution` — on Agent-run `git checkout`/`switch`/`worktree add`, bootstraps/maintains tokensave branch tracking (`branch add`); on other state-changing git commands (pull/merge/rebase/reset/cherry-pick/stash apply|pop), runs a coalesced incremental sync.
- `workspaceOpen` — ensures the current branch's DB exists (branch add if missing) and runs a catch-up incremental sync.
Expand Down Expand Up @@ -254,15 +254,101 @@ See [docs/BRANCHING-USER-GUIDE.md](docs/BRANCHING-USER-GUIDE.md) for the full gu

## Cross-Session Memory

Three MCP tools persist decisions and code-area context across sessions, stored in the per-project `.tokensave/tokensave.db`.
tokensave memory is stored in the per-project `.tokensave/tokensave.db` as durable, entity-linked facts:

| Tool | Purpose |
|------|---------|
| `tokensave_record_decision` | Save a design/architecture decision with optional reason, files, and tags |
| `tokensave_record_code_area` | Mark a path the agent has worked in (touch counter + last_touched_at) |
| `tokensave_session_recall` | FTS5 query over saved decisions; pair with the two write tools |
| `tokensave_fact_store` | Store a fact with entities, source, reason, related facts, contradictions, tags, and an initial confidence signal |
| `tokensave_fact_feedback` | Record user or agent feedback that raises, lowers, supersedes, or contradicts a fact's trust score |
| `tokensave_memory_status` | Inspect fact-store readiness, schema version, entity counts, trust-score distribution, and vector/backfill health |

Use these so the agent doesn't have to re-explain architecture choices session-to-session.
Trust scoring is fact-level, not just text-level. The store combines source metadata, feedback history, retrieval counters, contradiction scans, and recency into the returned score components. Entity recall returns facts linked to the requested symbol, file, subsystem, or named concept with `why` metadata so agents can explain why a memory was surfaced.

### Fact-store JSON schemas

`tokensave_fact_store` request schema:

```json
{
"type": "object",
"properties": {
"action": {
"type": "string",
"enum": ["add", "search", "probe", "related", "reason", "contradict", "update", "remove", "list"],
"description": "Fact-store action to perform."
},
"content": {
"type": "string",
"description": "Durable fact statement for add/update actions."
},
"query": {
"type": "string",
"description": "Search query for search actions."
},
"entity": { "type": "string" },
"entities": {
"type": "array",
"items": { "type": "string" }
},
"fact_id": { "oneOf": [{ "type": "number" }, { "type": "string" }] },
"category": { "type": "string", "enum": ["general", "user_pref", "project", "tool", "decision", "code_area"] },
"source": {
"type": "string",
"description": "Where the fact came from."
},
"trust": {
"type": "number",
"minimum": 0,
"maximum": 1,
"description": "Initial or replacement trust score."
},
"trust_delta": { "type": "number" },
"threshold": { "type": "number" },
"limit": { "type": "number" },
"metadata": { "type": "object" },
"tags": {
"type": "array",
"items": { "type": "string" }
}
},
"required": ["action"]
}
```

`tokensave_fact_feedback` request schema:

```json
{
"type": "object",
"properties": {
"fact_id": {
"oneOf": [{ "type": "number" }, { "type": "string" }],
"description": "Numeric fact_id returned by tokensave_fact_store or recall. Numeric strings are accepted."
},
"action": {
"type": "string",
"enum": ["helpful", "unhelpful"],
"description": "Feedback event to apply to the fact's trust score."
},
"source": {
"type": "string",
"description": "Optional feedback source label."
},
"note": {
"type": "string",
"description": "Optional note explaining the feedback."
},
"trust_delta": {
"type": "number",
"description": "Compatibility field; built-in helpful/unhelpful deltas are applied."
}
},
"required": ["fact_id"]
}
```

Provide either `"action": "helpful"|"unhelpful"` or the compatibility shorthand
`"helpful": true` / `"unhelpful": true`.

---

Expand Down
6 changes: 3 additions & 3 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,12 @@ tokensave builds a **local** code graph stored in a SQLite (libSQL) database (`.
- File paths, sizes, and content hashes
- Call relationships and dependency edges
- FTS5 search index
- Cross-session memory (recorded decisions and code-area notes)
- Cross-session memory: durable facts, named entities, code-area notes, decisions, and feedback events in the holographic fact store. Those rows are local-only project data.
- A response cache for `tokensave_read` (`read_cache` table): the rendered output served to the agent, stored as a BLOB keyed by file path, mode, and arguments. For full/line-range reads this rendered output contains source text. Rows are freshness-gated by file mtime and swept after a period of inactivity.

Aside from the `read_cache`, the graph itself does **not** persist raw source code — it stores structural metadata only. The database is local-only — there is no cloud sync, remote database, or server-side storage.

A second database (`~/.tokensave/global.db`) tracks which projects have been indexed, aggregate token-saved counts, and cost accounting data parsed from Claude Code session transcripts. It contains directory paths, counters, per-turn cost/token/category records, and JSONL parse offsets. No source code or conversation content is stored.
A second database (`~/.tokensave/global.db`) tracks which projects have been indexed, aggregate token-saved counts, and cost accounting data parsed from Claude Code session transcripts. Project-local Cursor transcript search is stored in the repository's `.tokensave/sessions.db`, which contains ingested Cursor user/assistant message text plus transcript paths and metadata for that project. Both databases remain local-only and are not synced to a remote service.

### Network access

Expand Down Expand Up @@ -72,7 +72,7 @@ The MCP server exposes **more than 70 tools** (one fewer when the optional `ast-
**Local-state tools** (write only inside `.tokensave/`, never your source):

- `tokensave_session_start`, `tokensave_session_end` — health-metric baselines
- `tokensave_record_decision`, `tokensave_record_code_area` — cross-session memory
- `tokensave_fact_store`, `tokensave_fact_feedback` — store fact text, entity names, feedback events, and trust-score inputs in the local project database. `tokensave_memory_status` is read-only.

**Test execution:**

Expand Down
2 changes: 1 addition & 1 deletion TEST-QUERIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -1074,4 +1074,4 @@ tokensave_field_sites() # missing field → error

---

> **Note:** Most tools are read-only and safe to call in parallel. The exceptions mutate state and should not be parallelised: the edit tools (`tokensave_str_replace`, `tokensave_multi_str_replace`, `tokensave_insert_at`, `tokensave_insert_at_symbol`, `tokensave_replace_symbol`, `tokensave_ast_grep_rewrite`) modify source files; the session and memory tools (`tokensave_session_start`, `tokensave_session_end`, `tokensave_record_decision`, `tokensave_record_code_area`) write to `.tokensave/`; and `tokensave_run_affected_tests` runs a `cargo` test subprocess.
> **Note:** Most tools are read-only and safe to call in parallel. The exceptions mutate state and should not be parallelised: the edit tools (`tokensave_str_replace`, `tokensave_multi_str_replace`, `tokensave_insert_at`, `tokensave_insert_at_symbol`, `tokensave_replace_symbol`, `tokensave_ast_grep_rewrite`) modify source files; the session and memory tools (`tokensave_session_start`, `tokensave_session_end`, `tokensave_fact_store`, `tokensave_fact_feedback`) write to `.tokensave/`; and `tokensave_run_affected_tests` runs a `cargo` test subprocess.
2 changes: 1 addition & 1 deletion benchmarks/tsbench/bench_tokensave.patch
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
+SYSTEM_PROMPT_TS = """You PREFER `mcp__tokensave__*` tools for code navigation and editing. NEVER spawn a sub-agent via Agent — do all work in the main session. Read / Grep / Glob / Edit / Write / Bash are ALLOWED only as a fallback when no tokensave tool covers the task (env-var orphan audits in `.env*` files, Dockerfile inspection, add-field-to-model on `.prisma`/`.ts`, multi-file symbol moves). For everything else, use `mcp__tokensave__*`.

-Active project: "tsbench" (preset — no switch_project needed). Do NOT call memory_search or memory_save.
+Active project: "tsbench" (indexed in-place — no init needed). Do NOT call any memory/recording tools (`tokensave_record_decision`, `tokensave_record_code_area`, `tokensave_session_recall`, `tokensave_session_start`, `tokensave_session_end`).
+Active project: "tsbench" (indexed in-place — no init needed). Do NOT call any memory/recording tools (`tokensave_fact_store`, `tokensave_fact_feedback`, `tokensave_session_start`, `tokensave_session_end`).

CORE
-- Locate: find_symbol(name, level=2). One call.
Expand Down
8 changes: 4 additions & 4 deletions docs/TOKENSAVE-VS-TOKENSAVIOR.md
Original file line number Diff line number Diff line change
Expand Up @@ -248,10 +248,10 @@ substitutes for batch lookups.
### 2.6 Cross-project memory

token-savior maintains a project-keyed memory store (`memory_search`,
`memory_save`) so notes from one project surface in another. tokensave has
narrower `tokensave_record_decision` / `tokensave_record_code_area` /
`tokensave_session_recall` tools but they're scoped to a single project
and not yet wired into a global recall mechanism.
`memory_save`) so notes from one project surface in another. tokensave uses
project-local holographic fact memory via `tokensave_fact_store`,
`tokensave_fact_feedback`, and `tokensave_memory_status`; cross-project memory
recall remains out of scope.

---

Expand Down
Loading
Loading