From a7fa2a40b8f4d753beedf317e84820eba4a6bc8b Mon Sep 17 00:00:00 2001 From: ScriptedAlchemy Date: Mon, 8 Jun 2026 01:04:58 +0200 Subject: [PATCH 1/2] docs(memory): document holographic memory and local installs --- README.md | 98 +++++++++++++++++++-- SECURITY.md | 6 +- TEST-QUERIES.md | 2 +- benchmarks/tsbench/bench_tokensave.patch | 2 +- docs/TOKENSAVE-VS-TOKENSAVIOR.md | 8 +- docs/USER-GUIDE.md | 103 +++++++++++++++++------ 6 files changed, 178 insertions(+), 41 deletions(-) diff --git a/README.md b/README.md index 2ea7e5622..ec1c0d225 100644 --- a/README.md +++ b/README.md @@ -73,7 +73,7 @@ AI coding agents waste tokens exploring codebases. Every grep, glob, and file re | **70+ MCP Tools** | **50+ Languages** | **12+ Agent Integrations** | | From call graph traversal to dead code detection, atomic edit primitives, code-health metrics, test mapping, and complexity analysis. | Rust, Go, Java, Python, TypeScript, C, C++, Swift, Svelte, Astro, and 42 more including WGSL/HLSL/Metal shaders and Markdown. Three tiers (lite/medium/full) control binary size. | Claude Code, Codex CLI, Gemini CLI, Hermes, Kiro, Cursor, OpenCode, Copilot, Cline, Roo Code, Zed, Antigravity, Kilo CLI, Kimi CLI, Mistral Vibe. | | **Multi-Branch Indexing (opt-in)** | **100% Local** | **Always Fresh** | -| Optional per-branch databases. Cross-branch diff and search without switching your checkout. | No data leaves your machine. No API keys. No external services. Everything runs on a local libSQL database. | On-demand staleness check on every MCP call (30 s cooldown) plus catch-up sync when the server connects. Multi-agent work is expected to use git worktrees — each agent gets its own checkout and the index diverges are merged by git, not by a file watcher. | +| Optional per-branch databases. Cross-branch diff and search without switching your checkout. | Source code and memory content stay on your machine. No API keys or hosted database are required; the index runs on local libSQL. | On-demand staleness check on every MCP call (30 s cooldown) plus catch-up sync when the server connects. Multi-agent work is expected to use git worktrees — each agent gets its own checkout and the index diverges are merged by git, not by a file watcher. | | **Subprocess-Isolated Extraction** | **Code-Health Analytics** | **Atomic Edit Primitives** | | A native crash in any tree-sitter grammar (abort, segfault, anything) kills only the worker; the pool respawns it and sync continues. Sync never dies on a malformed file. | Composite health score (0-10000), Gini inequality, file-DAG depth, design-structure matrix, risk-weighted test gaps, and session deltas. | Edit files without regex or shell-quoting hazards: unique-anchor `str_replace`, atomic multi-replace, AST-rewrite, anchored insert. Auto re-indexes after writes. | @@ -254,15 +254,101 @@ See [docs/BRANCHING-USER-GUIDE.md](docs/BRANCHING-USER-GUIDE.md) for the full gu ## Cross-Session Memory -Three MCP tools persist decisions and code-area context across sessions, stored in the per-project `.tokensave/tokensave.db`. +tokensave memory is stored in the per-project `.tokensave/tokensave.db` as durable, entity-linked facts: | Tool | Purpose | |------|---------| -| `tokensave_record_decision` | Save a design/architecture decision with optional reason, files, and tags | -| `tokensave_record_code_area` | Mark a path the agent has worked in (touch counter + last_touched_at) | -| `tokensave_session_recall` | FTS5 query over saved decisions; pair with the two write tools | +| `tokensave_fact_store` | Store a fact with entities, source, reason, related facts, contradictions, tags, and an initial confidence signal | +| `tokensave_fact_feedback` | Record user or agent feedback that raises, lowers, supersedes, or contradicts a fact's trust score | +| `tokensave_memory_status` | Inspect fact-store readiness, schema version, entity counts, trust-score distribution, and vector/backfill health | -Use these so the agent doesn't have to re-explain architecture choices session-to-session. +Trust scoring is fact-level, not just text-level. The store combines source metadata, feedback history, retrieval counters, contradiction scans, and recency into the returned score components. Entity recall returns facts linked to the requested symbol, file, subsystem, or named concept with `why` metadata so agents can explain why a memory was surfaced. + +### Fact-store JSON schemas + +`tokensave_fact_store` request schema: + +```json +{ + "type": "object", + "properties": { + "action": { + "type": "string", + "enum": ["add", "search", "probe", "related", "reason", "contradict", "update", "remove", "list"], + "description": "Fact-store action to perform." + }, + "content": { + "type": "string", + "description": "Durable fact statement for add/update actions." + }, + "query": { + "type": "string", + "description": "Search query for search actions." + }, + "entity": { "type": "string" }, + "entities": { + "type": "array", + "items": { "type": "string" } + }, + "fact_id": { "oneOf": [{ "type": "number" }, { "type": "string" }] }, + "category": { "type": "string", "enum": ["general", "user_pref", "project", "tool", "decision", "code_area"] }, + "source": { + "type": "string", + "description": "Where the fact came from." + }, + "trust": { + "type": "number", + "minimum": 0, + "maximum": 1, + "description": "Initial or replacement trust score." + }, + "trust_delta": { "type": "number" }, + "threshold": { "type": "number" }, + "limit": { "type": "number" }, + "metadata": { "type": "object" }, + "tags": { + "type": "array", + "items": { "type": "string" } + } + }, + "required": ["action"] +} +``` + +`tokensave_fact_feedback` request schema: + +```json +{ + "type": "object", + "properties": { + "fact_id": { + "oneOf": [{ "type": "number" }, { "type": "string" }], + "description": "Numeric fact_id returned by tokensave_fact_store or recall. Numeric strings are accepted." + }, + "action": { + "type": "string", + "enum": ["helpful", "unhelpful"], + "description": "Feedback event to apply to the fact's trust score." + }, + "source": { + "type": "string", + "description": "Optional feedback source label." + }, + "note": { + "type": "string", + "description": "Optional note explaining the feedback." + }, + "trust_delta": { + "type": "number", + "description": "Compatibility field; built-in helpful/unhelpful deltas are applied." + } + }, + "required": ["fact_id"] +} +``` + +Provide either `"action": "helpful"|"unhelpful"` or the compatibility shorthand +`"helpful": true` / `"unhelpful": true`. --- diff --git a/SECURITY.md b/SECURITY.md index 0eb5c20d6..b4a42dcc9 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -32,12 +32,12 @@ tokensave builds a **local** code graph stored in a SQLite (libSQL) database (`. - File paths, sizes, and content hashes - Call relationships and dependency edges - FTS5 search index -- Cross-session memory (recorded decisions and code-area notes) +- Cross-session memory: durable facts, named entities, code-area notes, decisions, and feedback events in the holographic fact store. Those rows are local-only project data. - A response cache for `tokensave_read` (`read_cache` table): the rendered output served to the agent, stored as a BLOB keyed by file path, mode, and arguments. For full/line-range reads this rendered output contains source text. Rows are freshness-gated by file mtime and swept after a period of inactivity. Aside from the `read_cache`, the graph itself does **not** persist raw source code — it stores structural metadata only. The database is local-only — there is no cloud sync, remote database, or server-side storage. -A second database (`~/.tokensave/global.db`) tracks which projects have been indexed, aggregate token-saved counts, and cost accounting data parsed from Claude Code session transcripts. It contains directory paths, counters, per-turn cost/token/category records, and JSONL parse offsets. No source code or conversation content is stored. +A second database (`~/.tokensave/global.db`) tracks which projects have been indexed, aggregate token-saved counts, and cost accounting data parsed from Claude Code session transcripts. Project-local Cursor transcript search is stored in the repository's `.tokensave/sessions.db`, which contains ingested Cursor user/assistant message text plus transcript paths and metadata for that project. Both databases remain local-only and are not synced to a remote service. ### Network access @@ -72,7 +72,7 @@ The MCP server exposes **more than 70 tools** (one fewer when the optional `ast- **Local-state tools** (write only inside `.tokensave/`, never your source): - `tokensave_session_start`, `tokensave_session_end` — health-metric baselines -- `tokensave_record_decision`, `tokensave_record_code_area` — cross-session memory +- `tokensave_fact_store`, `tokensave_fact_feedback` — store fact text, entity names, feedback events, and trust-score inputs in the local project database. `tokensave_memory_status` is read-only. **Test execution:** diff --git a/TEST-QUERIES.md b/TEST-QUERIES.md index aab55c1b9..c449e2fda 100644 --- a/TEST-QUERIES.md +++ b/TEST-QUERIES.md @@ -1074,4 +1074,4 @@ tokensave_field_sites() # missing field → error --- -> **Note:** Most tools are read-only and safe to call in parallel. The exceptions mutate state and should not be parallelised: the edit tools (`tokensave_str_replace`, `tokensave_multi_str_replace`, `tokensave_insert_at`, `tokensave_insert_at_symbol`, `tokensave_replace_symbol`, `tokensave_ast_grep_rewrite`) modify source files; the session and memory tools (`tokensave_session_start`, `tokensave_session_end`, `tokensave_record_decision`, `tokensave_record_code_area`) write to `.tokensave/`; and `tokensave_run_affected_tests` runs a `cargo` test subprocess. +> **Note:** Most tools are read-only and safe to call in parallel. The exceptions mutate state and should not be parallelised: the edit tools (`tokensave_str_replace`, `tokensave_multi_str_replace`, `tokensave_insert_at`, `tokensave_insert_at_symbol`, `tokensave_replace_symbol`, `tokensave_ast_grep_rewrite`) modify source files; the session and memory tools (`tokensave_session_start`, `tokensave_session_end`, `tokensave_fact_store`, `tokensave_fact_feedback`) write to `.tokensave/`; and `tokensave_run_affected_tests` runs a `cargo` test subprocess. diff --git a/benchmarks/tsbench/bench_tokensave.patch b/benchmarks/tsbench/bench_tokensave.patch index fd6a6b7ec..0d6e9cb1b 100644 --- a/benchmarks/tsbench/bench_tokensave.patch +++ b/benchmarks/tsbench/bench_tokensave.patch @@ -20,7 +20,7 @@ +SYSTEM_PROMPT_TS = """You PREFER `mcp__tokensave__*` tools for code navigation and editing. NEVER spawn a sub-agent via Agent — do all work in the main session. Read / Grep / Glob / Edit / Write / Bash are ALLOWED only as a fallback when no tokensave tool covers the task (env-var orphan audits in `.env*` files, Dockerfile inspection, add-field-to-model on `.prisma`/`.ts`, multi-file symbol moves). For everything else, use `mcp__tokensave__*`. -Active project: "tsbench" (preset — no switch_project needed). Do NOT call memory_search or memory_save. -+Active project: "tsbench" (indexed in-place — no init needed). Do NOT call any memory/recording tools (`tokensave_record_decision`, `tokensave_record_code_area`, `tokensave_session_recall`, `tokensave_session_start`, `tokensave_session_end`). ++Active project: "tsbench" (indexed in-place — no init needed). Do NOT call any memory/recording tools (`tokensave_fact_store`, `tokensave_fact_feedback`, `tokensave_session_start`, `tokensave_session_end`). CORE -- Locate: find_symbol(name, level=2). One call. diff --git a/docs/TOKENSAVE-VS-TOKENSAVIOR.md b/docs/TOKENSAVE-VS-TOKENSAVIOR.md index 0e0e33b36..596d78cf9 100644 --- a/docs/TOKENSAVE-VS-TOKENSAVIOR.md +++ b/docs/TOKENSAVE-VS-TOKENSAVIOR.md @@ -248,10 +248,10 @@ substitutes for batch lookups. ### 2.6 Cross-project memory token-savior maintains a project-keyed memory store (`memory_search`, -`memory_save`) so notes from one project surface in another. tokensave has -narrower `tokensave_record_decision` / `tokensave_record_code_area` / -`tokensave_session_recall` tools but they're scoped to a single project -and not yet wired into a global recall mechanism. +`memory_save`) so notes from one project surface in another. tokensave uses +project-local holographic fact memory via `tokensave_fact_store`, +`tokensave_fact_feedback`, and `tokensave_memory_status`; cross-project memory +recall remains out of scope. --- diff --git a/docs/USER-GUIDE.md b/docs/USER-GUIDE.md index f3710c188..3234e0c2a 100644 --- a/docs/USER-GUIDE.md +++ b/docs/USER-GUIDE.md @@ -402,32 +402,19 @@ cp scripts/post-commit .git/hooks/post-commit chmod +x .git/hooks/post-commit ``` -### Embedded file watcher +### MCP staleness checks -When you start the tokensave MCP server (e.g. via your agent), it watches the project directory and syncs automatically. See the next section. +When you start the tokensave MCP server (e.g. via your agent), tool calls perform an on-demand staleness check and catch up changed files before returning results. There is no long-running file watcher process. --- -## The Embedded File Watcher +## MCP Staleness Checks -When you start the tokensave MCP server (e.g. via your agent), it watches -the project directory for file changes and automatically runs incremental -syncs in the background. The watcher's lifetime is bound to the MCP -process — when the agent exits, the watcher exits. - -Multiple MCP servers on the same project (e.g. two agents) coordinate via -a per-project sync lock: only one sync runs at a time. - -Configure the debounce interval in `~/.tokensave/config.toml`: - -```toml -watcher_debounce = "15s" -``` +The MCP server does not run a background file watcher. Instead, MCP tool calls perform a lightweight staleness check and run an incremental sync when indexed files are stale. Multiple MCP servers on the same project coordinate via a per-project sync lock: only one sync runs at a time. ### CLI-Only Workflows -If you don't keep an agent attached, the watcher is not running. Use a -git post-commit hook to refresh the index on commit: +If you don't keep an agent attached, use a git post-commit hook to refresh the index on commit: ```bash cp scripts/post-commit .git/hooks/post-commit @@ -454,7 +441,7 @@ Then remove the entry matching your install: - Linux: `systemctl --user disable --now tokensave-daemon && rm ~/.config/systemd/user/tokensave-daemon.service` - Windows: `sc.exe delete tokensave-daemon` (from an elevated terminal) -Once your agent is attached, the embedded watcher takes over automatically. +Once your agent is attached, MCP tool calls keep the index fresh on demand. --- @@ -520,7 +507,7 @@ tokensave affected src/lib.rs --quiet # just file paths, no decoratio ## MCP Tools for AI Agents -When running as an MCP server, tokensave exposes 41 tools that AI agents can call. Here's what they do, grouped by purpose. +When running as an MCP server, tokensave exposes more than 70 tools that AI agents can call. Here's what they do, grouped by purpose. ### Core exploration @@ -618,27 +605,91 @@ Marked functions are excluded from `tokensave_test_risk` coverage calculations, | `tokensave_session_start` | Save current health metrics as a baseline before starting work. | | `tokensave_session_end` | Compare current health against the baseline to detect structural degradation during the session. | -Discovery and analysis tools are read-only and safe to call in parallel. Session baseline tools write/remove `.tokensave/session_baseline.json`, memory-recording tools update the project database, and edit tools modify source files. +### Memory and fact recall + +The holographic memory tools store durable facts linked to entities: + +| Tool | What it does | +|------|--------------| +| `tokensave_fact_store` | Store, search, update, remove, and reason over facts linked to entities such as symbols, files, branches, subsystems, people, or concepts. | +| `tokensave_fact_feedback` | Record `helpful` or `unhelpful` feedback for a numeric `fact_id` so the fact's computed trust score changes over time. | +| `tokensave_memory_status` | Report fact/entity counts, four trust-score buckets, feedback counts, and missing-vector count. | + +Entity recall should surface facts by named entity and include why each fact was recalled: matching entities, reason text, related fact IDs, contradiction links, and the current trust score. The legacy tools should remain as wrappers after integration so existing agent prompts and permissions keep working while new agents move to the fact-store surface. + +`tokensave_fact_store` request schema: + +```json +{ + "type": "object", + "properties": { + "action": { + "type": "string", + "enum": ["add", "search", "probe", "related", "reason", "contradict", "update", "remove", "list"] + }, + "content": { "type": "string" }, + "query": { "type": "string" }, + "entity": { "type": "string" }, + "entities": { + "type": "array", + "items": { "type": "string" } + }, + "fact_id": { "oneOf": [{ "type": "number" }, { "type": "string" }] }, + "category": { "type": "string" }, + "source": { "type": "string" }, + "trust": { "type": "number", "minimum": 0, "maximum": 1 }, + "trust_delta": { "type": "number" }, + "threshold": { "type": "number" }, + "limit": { "type": "number" }, + "metadata": { "type": "object" }, + "tags": { "type": "array", "items": { "type": "string" } } + }, + "required": ["action"] +} +``` + +`tokensave_fact_feedback` request schema: + +```json +{ + "type": "object", + "properties": { + "fact_id": { "oneOf": [{ "type": "number" }, { "type": "string" }] }, + "action": { + "type": "string", + "enum": ["helpful", "unhelpful"] + }, + "source": { "type": "string" }, + "note": { "type": "string" }, + "trust_delta": { "type": "number" } + }, + "required": ["fact_id"] +} +``` + +Provide either `"action": "helpful"|"unhelpful"` or the compatibility shorthand `"helpful": true` / `"unhelpful": true`. + +Discovery and analysis tools are read-only and safe to call in parallel. Session baseline tools write/remove `.tokensave/session_baseline.json`, memory-recording and future fact-feedback tools update the project database, and edit tools modify source files. --- ## Supported Languages -Tokensave supports 31 languages, organized into three tiers. Each tier includes all the languages from the tier below it. +Tokensave supports more than 50 languages, organized into three tiers. Each tier includes all the languages from the tier below it. -### Lite (11 languages) +### Lite (14 languages) Always compiled. The smallest binary for the most popular languages. -Rust, Go, Java, Scala, TypeScript, JavaScript, Python, C, C++, Kotlin, C#, Swift +Rust, Go, Java, Scala, TypeScript, JavaScript, Python, C, C++, Kotlin, C#, Swift, Svelte, Astro -### Medium (Lite + 9 = 20 languages) +### Medium (Lite + 9 = 23 languages) Adds scripting, config, and additional systems languages. Dart, Pascal, PHP, Ruby, Bash, Protobuf, PowerShell, Nix, VB.NET -### Full (Medium + 11 = 31 languages) +### Full (Medium + 27+ languages) Everything, including legacy and niche languages. From 81358dc45de46ff0b73d3d9bb754b3b7456daf38 Mon Sep 17 00:00:00 2001 From: ScriptedAlchemy Date: Mon, 8 Jun 2026 01:18:07 +0200 Subject: [PATCH 2/2] fix(docs): remove stale legacy memory guidance --- README.md | 4 ++-- docs/USER-GUIDE.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index ec1c0d225..5180bbd65 100644 --- a/README.md +++ b/README.md @@ -147,11 +147,11 @@ For project-scoped setup, run from the repository root: tokensave install --local --agent cursor ``` -Local install writes only workspace files such as `.cursor/mcp.json`, `.mcp.json`, `.codex/config.toml`, `.vscode/mcp.json`, `.hermes/plugins/tokensave/`, or the equivalent project config for Claude, Codex, Gemini, Hermes, Kiro, OpenCode, Copilot/VS Code, Zed, Roo Code, Kimi, Kilo, and Vibe. Generated MCP configs and plugin wrappers use the resolved absolute `tokensave` executable path. Hermes installs into `~/.hermes/plugins/tokensave/` by default, or into `~/.hermes/profiles//plugins/tokensave/` with `--profile `; profile names are normalized to lowercase and must match `[a-z0-9][a-z0-9_-]{0,63}`. Use `tokensave uninstall --agent hermes --profile ` to remove a named profile install; `reinstall` and `doctor --agent hermes` currently operate on the default profile. Hermes wrappers run from Hermes' current working directory, use a 600-second timeout, and include truncated stdout/stderr in error JSON. Hermes local install without `--profile` writes only project plugin files and `.hermes/config.yaml`; `tokensave install --local --agent hermes --profile ` is a deliberate mixed-scope mode that targets the named profile instead. Hermes requires `HERMES_ENABLE_PROJECT_PLUGINS=true` when launching with project-local plugins. For Cursor, local install also writes `.cursor/rules/tokensave.mdc`, `.cursor/permissions.json`, and `.cursor/hooks.json`: the rule tells Cursor Agent to prefer tokensave MCP tools for codebase exploration, and permissions auto-allow only read-only tokensave MCP tools. The project hooks are: +Local install writes only workspace files such as `.cursor/mcp.json`, `.mcp.json`, `.codex/config.toml`, `.vscode/mcp.json`, `.hermes/plugins/tokensave/`, or the equivalent project config for Claude, Codex, Gemini, Hermes, Kiro, OpenCode, Copilot/VS Code, Zed, Roo Code, Kimi, Kilo, and Vibe. Generated MCP configs and plugin wrappers use the resolved absolute `tokensave` executable path. Hermes installs into `~/.hermes/plugins/tokensave/` by default, or into `~/.hermes/profiles//plugins/tokensave/` with `--profile `; profile names are normalized to lowercase and must match `[a-z0-9][a-z0-9_-]{0,63}`. Use `tokensave uninstall --agent hermes --profile ` to remove a named profile install; `reinstall` and `doctor --agent hermes` currently operate on the default profile. Hermes wrappers run from Hermes' current working directory, use a 600-second timeout, and include truncated stdout/stderr in error JSON. Hermes local install without `--profile` writes only project plugin files and `.hermes/config.yaml`; `tokensave install --local --agent hermes --profile ` is a deliberate mixed-scope mode that targets the named profile instead. Hermes requires `HERMES_ENABLE_PROJECT_PLUGINS=true` when launching with project-local plugins. For Cursor, local install also writes `.cursor/rules/tokensave.mdc`, `.cursor/permissions.json`, and `.cursor/hooks.json`: the rule tells Cursor Agent to prefer tokensave MCP tools for codebase exploration, and permissions auto-allow the full local tokensave MCP surface for that workspace. The project hooks are: - `sessionStart` — fire-and-forget; injects context steering the Agent toward tokensave MCP tools and reports index freshness (suggests `tokensave init` when no `.tokensave/` exists). - `subagentStart` — blocks research/explore subagents until tokensave MCP tools have been tried. -- `beforeSubmitPrompt` — resets the local token counter for the new turn. +- `beforeSubmitPrompt` — resets the local token counter for the new turn and ingests the current Cursor transcript into `.tokensave/sessions.db` when `transcript_path` is present. - `afterFileEdit` (matcher `Write`) — runs a **targeted single-file** sync of just the edited path(s) via `sync_if_stale_silent`, never a full-tree scan (which would scale with repo size, not edit size). - `afterShellExecution` — on Agent-run `git checkout`/`switch`/`worktree add`, bootstraps/maintains tokensave branch tracking (`branch add`); on other state-changing git commands (pull/merge/rebase/reset/cherry-pick/stash apply|pop), runs a coalesced incremental sync. - `workspaceOpen` — ensures the current branch's DB exists (branch add if missing) and runs a catch-up incremental sync. diff --git a/docs/USER-GUIDE.md b/docs/USER-GUIDE.md index 3234e0c2a..d26a649ae 100644 --- a/docs/USER-GUIDE.md +++ b/docs/USER-GUIDE.md @@ -615,7 +615,7 @@ The holographic memory tools store durable facts linked to entities: | `tokensave_fact_feedback` | Record `helpful` or `unhelpful` feedback for a numeric `fact_id` so the fact's computed trust score changes over time. | | `tokensave_memory_status` | Report fact/entity counts, four trust-score buckets, feedback counts, and missing-vector count. | -Entity recall should surface facts by named entity and include why each fact was recalled: matching entities, reason text, related fact IDs, contradiction links, and the current trust score. The legacy tools should remain as wrappers after integration so existing agent prompts and permissions keep working while new agents move to the fact-store surface. +Entity recall surfaces facts by named entity and includes why each fact was recalled: matching entities, reason text, related fact IDs, contradiction links, and the current trust score. The legacy memory tools are no longer exposed; update old prompts and permissions to use `tokensave_fact_store`, `tokensave_fact_feedback`, and `tokensave_memory_status`. `tokensave_fact_store` request schema: