Codebase Index

Codebase Index

Index, search, and trace the structural entities of your project source code — instead of reading files sequentially to understand a codebase, agents can query a pre-built index that provides structured symbol information on demand.

What is Codebase Index?

The Codebase Index is a source code analysis pipeline integrated into the MCP server. It:

1. Discovers source files in a repository directory
2. Parses them using tree-sitter (WASM) to extract structural symbols
3. Stores the results in SQLite tables alongside existing memory data
4. Queries them through 2 unified MCP tools (codebase-index, codebase-read) that agents can call at any time

The result is a searchable, structured view of your codebase that persists across sessions and updates incrementally. Design rationale: ADR-002 (Codebase Index Architecture).

How It Works

The indexing pipeline operates in 5 phases: Discover → Compare → Parse → Store → Clean.

Phase 1: DISCOVER

Walks the repository directory tree using fast-glob. Automatically respects .gitignore rules and detects language by file extension. Default exclusions prevent indexing of build artifacts and dependency directories: node_modules, .git, dist, .next, build, coverage, __pycache__, .venv, vendor, target, .DS_Store. Custom include/exclude glob patterns can be provided per indexing run.

Phase 2: COMPARE

For each discovered file, the system compares its mtime against the stored last_indexed_at. Files whose mtime is more than 2000ms (MTIME_AMBIGUITY_MARGIN_MS) before their last index are skipped without being read — this mtime pre-filter is the incremental indexing mechanism. Files that are new, or whose mtime falls inside the ambiguity window, fall through to read + checksum confirmation in the parse phase, so a quick edit is never falsely skipped.

Phase 3: PARSE

Changed/new candidates run through a batched loop (runParsePipeline), with each batch capped at the parser concurrency (4 by default):

1. Read + checksum — file content is read and its SHA-256 checksum computed, without parsing. Files larger than 10MB are rejected here and marked as failed (no timeout fallback for oversized files).
2. Sequential decision — touch-only files (checksum unchanged) are skipped, renames are detected by matching checksums against now-stale paths, and only genuinely changed/new files move on.
3. Parse only changed files — each surviving file is parsed with tree-sitter WASM (10-50ms per file) and traversed by a language-specific visitor to extract functions, methods, classes, interfaces, type aliases, enums, and module-scope variables.

Each parse has a 10-second timeout — files exceeding it are marked as failed. File and symbol inserts are flushed to the database after every batch, so memory stays bounded regardless of repository size.

Phase 4: STORE

Under a write lock: file records are upserted (path, language, checksum, line count, size), old symbol records for re-indexed files are deleted, and new symbol records are bulk-inserted. The FTS5 virtual table (codebase_symbols_fts) is kept in sync automatically via INSERT/UPDATE/DELETE triggers.

Phase 5: CLEAN

Database records for files that no longer exist on disk are removed, keeping the index consistent with the actual filesystem state.

Supported Languages

Language	File Extensions	Status	Reference edges
TypeScript	.ts, .tsx, .mts, .cts, .svelte, .astro	✅ Full	call · instantiation · import · extends · implements
JavaScript	.js, .jsx, .mjs, .cjs	✅ Full	call · instantiation · import · extends · implements
Vue	.vue	✅ Full	import · instantiation (template component tags)
Go	.go	✅ Full	call · import · extends
Python	.py	✅ Full	call · import · extends
PHP	.php	✅ Full	call · instantiation · import · extends · implements
Rust	.rs	✅ Full	call · import · extends · implements
Java	.java	✅ Full	call · import · extends · implements
Dart	.dart	✅ Full*	call · import · extends · implements
Kotlin	.kt, .kts	✅ Full	call · import · extends · implements
Ruby	.rb	✅ Full	call · import · extends
Swift	.swift	✅ Full	call · import · extends · implements
C	.c, .h	✅ Full	call · import
C++	.cpp, .cc, .cxx, .hpp, .hh, .hxx	✅ Full	call · import · extends · implements
Markdown	.md, .mdx	✅ Full	— (declarations only)

* Dart requires a compatible tree-sitter grammar WASM — see ABI compatibility notes in the operational guide.

15 languages are registered plus a generic catch-all config (16 configs total); 14 are parsed through tree-sitter grammars, Markdown uses a dedicated visitor without WASM, and a generic text visitor covers every other extension (JSON, YAML, CSS, shell scripts, etc.). Since Phase 1.1, 14 of the 16 configs emit reference edges (call, instantiation, import, extends, implements — the coverage varies per language; Markdown and the generic catch-all are declarations-only by design).

How to Use

CLI: --index Flag

The server can be invoked with the --index flag to perform a one-time indexing operation without starting the MCP server:

local-memory-mcp --index --repo owner/repo --path /absolute/path/to/repo

With glob filters:

local-memory-mcp --index \
  --repo owner/repo \
  --path /home/user/projects/my-app \
  --include "src/**/*.ts" \
  --exclude "**/*.test.ts" \
  --exclude "**/*.spec.ts"

Progress is printed to stderr with timestamps. Exit code is 0 on success, 1 on failure.

CLI flags:

Flag	Required	Repeatable	Description
--repo	Yes	No	Repository identifier (owner/repo).
--path	Yes	No	Absolute filesystem path to the repository.
--include	No	Yes	Glob pattern to include (e.g., src/**/*.ts).
--exclude	No	Yes	Glob pattern to exclude.

MCP Tools

The Codebase Index exposes 2 unified MCP tools (mode auto-inferred from parameters):

Tool	Modes (auto-inferred)	Description
codebase-index	INDEX, STATUS	Index/re-index a repository, or check its status.
codebase-read	TRACE, FILE, SEARCH, ARCHITECTURE, CODE	Read-only queries of the index, including content grep.

Legacy names are historical notes only — the pre-unification tool names (index_repository, index_status, search_symbols, get_file_symbols, get_architecture, trace_symbol, codebase_search) are NOT resolvable aliases; always call the canonical names. search_code (content grep with symbol context) was design intent only — it never shipped as a tool; the feature exists as the CODE mode of codebase-read. Full input/output schemas live in the developer docs (.agents/documents/api/codebase-index.md).

Dashboard

Navigate to the Codebase Index tab to see which repositories have been indexed, file and symbol counts per repository, and the language breakdown. The dashboard integration is read-only for now — indexing is triggered via CLI, MCP tools, or the startup auto-index. See the Dashboard Guide.

Performance Characteristics

Metric	Typical Value
First-time index (10K files)	~30-60 seconds
Incremental re-index	~1-5 seconds (only changed files parsed; an unchanged repo parses 0 files)
File parsing (per file)	10-50ms (tree-sitter WASM)
Query response (read)	<100ms for projects up to 20K files
WASM initialization	~500ms-1s (loaded once, cached)
Database growth (10K files)	~10-50MB additional

codebase-index (INDEX mode) does not hold the write lock during the heavy scan/parse work — the indexing writer acquires the lock per database batch. codebase-read is read-only and runs without blocking.

Database Schema

The Codebase Index uses three tables in the existing memory.db (plus the FTS5 virtual table):

• codebase_files — file records: id, repo, file_path, language, checksum (SHA-256), lines, size_bytes, last_indexed_at. Unique index on (repo, file_path).
• codebase_symbols — symbol records: id, repo, file_path, name, kind (function, class, interface, type, enum, method, variable), exported, default_export, start/end line+col, signature, doc_comment, parent_symbol_id. Indexed on (repo, name), (repo, file), (repo, kind), (name), (parent).
• codebase_symbols_fts — FTS5 full-text search over name, doc_comment, and signature, auto-synchronized via triggers.
• codebase_references — one row per reference edge: symbol_name (callee / base class / implemented interface), caller_file, caller_line, caller_name, kind (call | instantiation | import | extends | implements), plus resolvable target_file / target_symbol_id. Indexed on (repo, symbol_name) and (repo, caller_file).

Resolution is name-based (ADR-002): target_file / target_symbol_id are populated when the referenced name resolves at parse time; unresolved names keep them NULL and resolution falls back to query-time name matching.

Known Limitations

• Name-based resolution only. Symbol tracing and reference detection work by exact name string matching across files — there is no type-graph resolution (tree-sitter provides syntactic trees, not semantic type information). Two same-name symbols in different scopes may conflate or require disambiguation.
• Content search (CODE mode) is disk grep, not FTS. It greps the indexed files on disk (codebase_files scope only — node_modules/.git/untracked files are excluded by construction), enriched with each match's enclosing symbol span. Content is read from the caller-supplied repoPath through a process-shared LRU cache keyed to the file checksum; regex: true is guarded against ReDoS-style patterns (length cap + nested-quantifier rejection → INVALID_REGEX).
• Dead-code & hotspots (ARCHITECTURE mode). When enabled, the ARCHITECTURE mode appends a deadCode block: unreferenced[] (zero-reference top-level symbols), hotspots[] (top in-degree symbols), and languageCoverage. Entry-point exclusion covers package.json bin/main/exports/browser, #! shebang, and exported top-level symbols; Markdown/generic languages are reported as declaration-only.
• Incremental refresh + polling file watcher. Indexing is triggered by the codebase-index tool or CLI --index (explicit), startup auto-index when the last index is older than 24h (CODEBASE_AUTO_INDEX / CODEBASE_AUTO_INDEX_TTL), and a polling file watcher (ENABLE_FILE_WATCHER, default on) sweeping all registered repos every 30s (FILE_WATCH_INTERVAL_MS), with per-repo re-entry capped at 5 min (FILE_WATCH_TTL_MS). The watcher is hosted by the MCP server process only — the dashboard does NOT host it (would double-index). It is a bounded-delay polling sweep, not a real-time notification (fs.watch); detection latency is up to the configured interval.
• Database growth. Indexing large projects adds to memory.db size (10K TypeScript files ≈ 10-50MB). The database uses WAL mode, so reads are not blocked during writes.

Related Pages

• Tool Reference & Usage Guide — codebase-index and codebase-read tools
• Web Dashboard Guide — the Codebase tab
• Core Features — feature overview