VectorMCP — central cross-repo code index over MCP

Gives Claude Code one central, always-warm index over every Ikshana repo. State a task, get back the repos responsible for it, then search and read their code without cloning anything.

The constraint this is built around

Claude Code clones a fresh copy of a repo for each task and throws it away. So index identity is git object identity, never filesystem location: a chunk's Qdrant point id is uuid5(NS, "{repo_id}:{blob_sha}:{chunk_idx}"). Blob shas are identical in every clone, forever, which means:

• a fresh clone at any commit needs zero re-indexing;
• indexing runs server-side against bare mirrors, never in the agent's checkout;
• the agent only ever contributes deltas for its uncommitted edits, into a disposable scope=session:<id> overlay.

Measured on this codebase: a single commit changes 0.5–0.8% of indexable files, so an incremental pass re-embeds one or two files rather than 927.

Layout

path	what
codeindex/ignore.py	what not to index — the highest-leverage file here
codeindex/gitmirror.py	git access; works on bare mirrors or existing checkouts
codeindex/scan.py	commit → manifest, plus the reviewable report
codeindex/db.py	Postgres: manifests, embedding cache, ACL
codeindex/vectors.py	Qdrant collections, point ids, visibility filters
codeindex/openrouter.py	embeddings, rerank, summarisation ("online mode")
codeindex/chunking.py	syntax-aware chunking; copes with a 593 KB single-class module
codeindex/lexical.py	code-aware sparse/BM25 tokens (snake_case, camelCase, dotted paths)
codeindex/indexer.py	blob → chunks → summaries → vectors → Qdrant, batched repo-wide
codeindex/search.py	hybrid fusion, cross-encoder rerank, clone-free file reads
codeindex/cards.py	per-repo routing cards, used as a routing prior
codeindex/routing.py	task → responsible repos, evidence, sparse-checkout plan
codeindex/session.py	overlays for the agent's uncommitted edits
codeindex/extract.py	tree-sitter extraction: symbols, imports, routes, env keys, calls, tasks
codeindex/graph.py	cross-repo coupling edges and context expansion
codeindex/mcp_server.py	the MCP tools
codeindex/webui.py	local inspection UI backend (Starlette, no new deps)
codeindex/ui.html	the UI itself; re-read per request, so edits are live
codeindex/cli.py	setup, add-repo, scan, sync-manifests, verify-contents, index, search, verify-index, serve-*
sql/01_schema.sql	manifests, graph slice, sessions
sql/06_org_token.sql	the token -> organisation mapping
sql/07_control.sql	repo credentials, webhook secrets, the job queue
codeindex/control.py	the control API: registration, webhooks, org lifecycle

Running it

cp .env.example .env          # then set OPENROUTER_API_KEY
docker compose up -d qdrant postgres
docker compose build indexer

Ports are deliberately non-default — 6335/6336 for Qdrant and 5433 for Postgres — because the Qdrant on 6333 holds live person_reid and face collections and this is a tool whose own collections get dropped and rebuilt.

The vector dimension is fixed at collection creation and OpenRouter does not document it per model, so it is measured:

python3 scripts/probe_embedding.py --rerank   # prints EMBED_DIM -> put it in .env
docker compose run --rm indexer python -m codeindex.cli setup --owner acme

setup applies the schema and creates one organisation's collections; run it per org. It is safe to re-run: the base schema is created once and the migrations after it are idempotent, which is how an existing database picks up a newly added one.

Bootstrap from the checkouts already on this machine (no Bitbucket credentials needed — LOCAL_REPO_ROOT is bind-mounted read-only at /repos):

docker compose run --rm indexer python -m codeindex.cli add-repo /repos/BACKEND
docker compose run --rm indexer python -m codeindex.cli sync-manifests

Check the ignore policy against real repos before spending anything on embeddings — this needs no Docker, no Postgres and no API key:

python3 scripts/scan_report.py ~/Documents/Ikshana-V2/*/

Organisations

Everything is scoped to an organisation, and Qdrant collections are per organisation: code_chunks__<org> holds that org's code and its documents, repo_cards__<org> its routing cards. CODE_COLLECTION is a name prefix, not a collection name.

Per-org rather than per-repo, because a repo is not a unit anything is ever dropped or rebuilt at, whereas an org is -- drop_org is two calls -- and because it bounds the blast radius of a mistake. Every query still filters on owner_id inside the collection, so a bug that opened the wrong collection returns nothing rather than another customer's source.

A request names its organisation with a bearer token; the server no longer infers one from its own config:

docker compose run --rm indexer python -m codeindex.cli token --mint "acme ci" --owner acme
docker compose run --rm indexer python -m codeindex.cli token            # list
docker compose run --rm indexer python -m codeindex.cli token --revoke vmcp_AbC123

Only the sha256 is stored, so the plaintext is shown once and a leaked database row cannot be replayed. REQUIRE_AUTH=false serves an unauthenticated caller as DEFAULT_OWNER_ID and exists so a local checkout works with no token; a wrong token is refused either way.

Registering repos from a product

add-repo takes a checkout on the indexer's own disk, which is fine for a bootstrap and useless from a product. The control plane is the other way in:

endpoint	what
POST /v1/orgs/{org}/repos	register (or update) a repo; creates the org's collections on first use and queues a sync
GET /v1/orgs/{org}/repos	what is registered, with commit, chunk count and job state
DELETE /v1/orgs/{org}/repos/{slug}	deregister, taking its vectors with it
POST /v1/orgs/{org}/repos/{slug}/reindex	queue a sync by hand
POST /v1/orgs/{org}/tokens	mint an MCP bearer token for the org
DELETE /v1/orgs/{org}	drop the organisation entirely
POST /hooks/{org}/{slug}	a push at the git host -> a queued sync

It runs on 8081, separate from the MCP endpoint on 8080, and takes a different credential (CONTROL_TOKEN) because it can delete an organisation's whole index. With CONTROL_TOKEN unset it refuses every request rather than running open.

/hooks/... is the one public route, so it authenticates per repo on an HMAC of the raw body against that repo's own secret -- checked before the body is parsed, and answering 401 for an unknown repo so the endpoint cannot be used to enumerate which repos exist. The secret is returned on every registration, not just the first, so a caller that lost it can reconfigure the git host without re-registering.

Queued work is coalesced: at most one outstanding job per repo, enforced by a partial unique index. Ten pushes in a minute cause one index run against the newest commit rather than ten against ten commits.

Credentials for a private repo are stored as a reference (credential_ref), not a token. A token copied here would outlive its revocation and would have to be re-copied on rotation, and there would then be two places it could leak from.

Using it from Claude Code

claude mcp add --transport http codeindex http://localhost:8080/mcp \
  --header "Authorization: Bearer $CODEINDEX_TOKEN"

The header is what tells the server which organisation you are; it can be omitted only against a local stack running REQUIRE_AUTH=false.

Tools, in the order an agent should reach for them:

tool	what it answers
route_task	which repos own this task, with evidence and a sparse-checkout plan
search_code	which chunks are relevant, across every repo at once
get_file	the contents of any file at any commit, without cloning
expand_context	what else a symbol touches — callers in other services, routes, config
sync_workspace	make your own uncommitted edits searchable
session_status / end_session	inspect and discard an overlay
list_repos	what is indexed

The UI

http://localhost:8090 — three views over the same data the MCP tools return, so what you see is what the agent sees.

view	what it shows
Index	per-repo files / indexed / chunks / points, summary coverage, commit. Click a repo for its routing card, language mix and skip reasons
Search	the full hybrid + rerank pipeline, with each hit's summary and code
Route	ranked repos with the score broken into peak / depth / spread / prior / substance, the evidence, and the checkout plan

The Index tab also shows the structural graph — extraction totals and the cross-repo coupling table. Open session overlays appear below it, and entering a session id on the Search tab includes that session's uncommitted edits — flagged uncommitted in the results. Repo point counts are scoped to main, so an overlay never inflates a repo's committed total.

Clicking any path:line-line opens the real file at that commit, read from git objects with 25 lines of context either side and absolute line numbers — so a reported range can be checked against the actual file rather than trusted.

Views are linkable: ?q=<query> runs a search (add &sess=<id> to include a session's edits), ?t=<task> routes a task, ?tab=index|search|route switches.

This is served by the stack rather than published anywhere, because it has to reach Qdrant, Postgres and OpenRouter on localhost.

Checks

docker compose run --rm indexer python -m codeindex.cli verify-contents  # reject blobs the bytes disqualify
docker compose run --rm indexer python -m codeindex.cli index            # safe to re-run; unchanged blobs hit the cache
docker compose run --rm indexer python -m codeindex.cli extract          # structural facts (~6s, pure function of the commit)
docker compose run --rm indexer python -m codeindex.cli edges            # derive cross-repo coupling
docker compose run --rm indexer python -m codeindex.cli verify-index     # reconcile manifest -> chunks -> points
docker compose run --rm indexer python scripts/eval_routing.py           # routing regression suite (9 hand-verified cases)
docker compose run --rm indexer python scripts/eval_session.py           # session overlay lifecycle (5 assertions)
docker compose run --rm indexer python scripts/eval_tenancy.py           # org isolation + token auth (31 assertions)
docker compose run --rm indexer python scripts/eval_control.py           # registration, webhook auth, deletion (34 assertions)

verify-index exits non-zero if any repo is short of points, and reconciles against repo_commit.chunk_count rather than the blob_chunk cache -- that cache is content-addressed and shared across repos and commits, so it accumulates and cannot serve as a baseline.

The structural graph

Vector search answers "what looks like this". It cannot answer "what else breaks if I change this", because that is a question about structure — and structure is deterministic, so tree-sitter gives it up for free with no model and no ambiguity. Extraction over all seven repos takes ~6 seconds:

symbols	6,253
calls	52,911
routes	476 (Django path()/re_path() and FastAPI/Flask decorators)
imports	4,260
env keys	336
celery tasks	25
URL literals	354

Calls are matched by name, not resolved to a definition. Resolution needs type inference; a name match over 800 Python files is cheap, has no failure modes, and answers the two questions that matter: who calls this, and which other repo is coupled to it. Model.objects.filter(...) also records the root of the dotted chain, so referencing a shared model counts as a usage rather than being invisible behind filter.

Coupling rules

Chosen for what is genuinely invisible to a reader — an import graph misses all of these:

kind	evidence
http_call	a route registered in one repo, hardcoded as a URL in another
celery_task	producer and consumer share only the task name
env_key	two services that must agree on a config key (skipped if >3 repos use it — that is a house convention, not a coupling)
db_model	the same top-level model class in two repos means a shared schema

route_task reports coupling filtered to the task: an edge is only listed if its evidence appears in the code the search actually surfaced. Every repo here is coupled to the largest one somehow, so listing a repo's couplings wholesale says nothing about the task at hand. Counts read 2 here of 9 — matched versus total — rather than overstating.

Session overlays

The server indexes commits, which is everything except the file the agent is currently editing. So the agent sends its dirty files to sync_workspace and they land in the same collection under scope=session:<id>:

sync_workspace(session_id="fix-alerts", repo="cloud-ikshana",
               files={"path.py": "<contents>"}, deleted=["gone.py"])
search_code(query="...", session_id="fix-alerts")   # sees the edits
end_session("fix-alerts")

Shadowing happens in the Qdrant filter, not after retrieval: paths the session has touched are excluded at main scope. Post-filtering cannot do this — a deleted file has no session points to shadow with, so it would keep returning its committed version forever.

Cost is proportional to what the agent touched. The other 5,400 chunks are already embedded and stay that way, which is the entire point: nothing an agent does can corrupt or re-cost the committed index. Overlays are dropped by end_session and swept after 24 idle hours by the indexer loop, so a crashed agent leaks nothing permanent.

Routing

route_task scores each repo on four signals and returns the ones that survive a relative floor:

signal	weight	why
peak	0.48	best chunk after cross-encoder rerank
depth	0.28	mean of the top 5, so one lucky match is not enough
spread	0.10	distinct files, saturating -- 35 files is not 4x the ownership of 8
prior	0.14	task vs the repo's card, for vocabulary matches with no standout chunk

Reranking happens before grouping, because RRF fusion scores are rank-based and not comparable across repos. The candidate pool is capped per repo, since one repo holds 78% of all chunks and would otherwise crowd the alternatives out of the reranker entirely. A final substance factor demotes repos whose evidence is all fixtures and docs -- as a threshold, not a ratio, so a small repo whose README genuinely explains its behaviour is not punished for that.

Retrieval design

Small corpus (~26k chunks), so quality wins over every efficiency trade:

• no quantization, and exact=True while under 250k points — brute force with perfect recall instead of an HNSW approximation;
• two dense vectors per point: the code itself, and an LLM-written description of it. Task text is natural language and matches the latter far better than it matches Django internals;
• a sparse lex vector with server-side IDF, because tasks name exact identifiers that dense vectors miss;
• cross-encoder rerank of the fused top ~150 down to ~20. Largest single precision gain in the pipeline.

Status

Phases 0-4 done and verified end to end. 5,467 chunks across 7 repos, 100% with an LLM summary, verify-index clean, routing 9/9 at rank 1, session overlays 5/5, and a structural graph of 6,253 symbols / 52,911 calls / 476 routes with 90 cross-repo coupling edges. Measured: hybrid fusion ~900 ms (including the query embedding round trip), cross-encoder rerank ~1.8 s, extraction ~6 s for everything.

Multi-org is in: collections are per organisation, and every MCP tool resolves its org from the caller's token instead of server config -- 31/31 in eval_tenancy, covering collection isolation, token lifecycle, and the fact that a grant on one org's repo cannot widen what another org sees. The tool schemas are unchanged, so existing clients see the same surface.

Registration is in too: Agent Studio registers a repo over the control API, which creates the org's collections, records the credential reference, and queues a sync -- 34/34 in eval_control, and verified end to end from the Agent Studio API through to Qdrant.

Next: serve-indexer becomes a real pipeline (fetch -> scan -> index delta -> extract -> cards) that drains the job queue, so a queued sync actually runs. Then documents alongside code, under kind=doc in the same per-org collection.

Credential material is excluded, and there is some to deal with

Indexing a secret copies it into a vector store and then into agent context on every loosely-related search, so credential material is rejected on the path where possible (serviceAccountKey.json, .env, *.pem, *.key) and on the bytes where not. Source files are handled at chunk granularity instead -- dropping all of settings.py or 25 chunks of backup_manager.py would cost real code, so the file is indexed and only the offending chunk is held back. Verified: 0 of 5,467 points contain credential markers.

What that turned up in the repos themselves is listed under Known gaps.