Phase 8: file-type-aware embedding model routing, dual-model search, and incremental indexing

[Copilot]

Different content types benefit from different embedding models — code-aware models (e.g. nomic-embed-code) capture syntax-rich vocabulary better than general text models. This adds a routing layer that selects the optimal model per blob based on file extension, dual-model search that queries both embedding spaces and merges results, and an incremental --since flag for the indexer.

New modules

• src/core/embedding/fileType.ts — classifies paths as 'code' | 'text' | 'other' via CODE_EXTENSIONS / TEXT_EXTENSIONS sets; exports getFileCategory(path)
• src/core/embedding/router.ts — RoutingProvider wraps two providers; providerForFile(path) picks the active one for indexing

Core changes

• IndexerOptions gains optional codeProvider; when set, RoutingProvider is used per blob; single-provider path unchanged
• IndexerOptions gains optional since; restricts which commits are indexed (see CLI section)
• embeddings schema gains nullable file_type column recording which category produced the embedding
• StoreBlobArgs accepts optional fileType and persists it in the transaction
• vectorSearch gains optional model filter and a new mergeSearchResults export for combining results from different embedding spaces

CLI / config

Two new env vars; default behaviour (single model) is unaffected:

# Dual-model indexing
GITSEMA_TEXT_MODEL=nomic-embed-text \
GITSEMA_CODE_MODEL=nomic-embed-code \
  gitsema index

# gitsema status now shows both models when they differ
# scripts/embed.ts accepts optional filePath to demo routing

When GITSEMA_CODE_MODEL differs from GITSEMA_TEXT_MODEL, gitsema search embeds the query with both models, runs two vector searches (each filtered to embeddings produced by the matching model), and merges results keeping the best score per blob.

# Incremental indexing — accepts a date, tag, or commit
gitsema index --since 2024-01-01   # date → --after= filter
gitsema index --since v1.2.0       # tag  → v1.2.0..HEAD range
gitsema index --since HEAD~100     # symbolic ref or commit hash

Plan

docs/PLAN.md gains a new Phase 8 section with the full design rationale; former Phase 8 (Performance) → 9, Smarter semantics → 10, Advanced + MCP → 11.

Original prompt

Do we have different models for different type of files? Code aware embed for code and text for txt and embed?

Please enhance plan. Add a phase after phase 7 (new 8) that implements two models. Plan this.

[jsilvanus]

@copilot implement phase 8.

[Copilot]

@copilot implement phase 8.

Phase 8 is fully implemented (cf39221). Here's what was built:

New modules

• src/core/embedding/fileType.ts — classifies any file path as code, text, or other by extension (30+ code extensions, markdown/rst/txt/etc. as text)
• src/core/embedding/router.ts — RoutingProvider wraps two providers and picks the right one per file; search queries always use the text provider

Core wiring

• IndexerOptions has optional codeProvider; when set, each blob is routed to its best-fit model; single-model behaviour unchanged
• embeddings table gains a nullable file_type column (both in Drizzle schema and the raw DDL); existing databases are auto-migrated on next startup via ALTER TABLE
• blobStore stores the fileType alongside each embedding

CLI

# Dual-model indexing
GITSEMA_TEXT_MODEL=nomic-embed-text \
GITSEMA_CODE_MODEL=nomic-embed-code \
  gitsema index

# gitsema status reports both models when they differ
# scripts/embed.ts <text> <filePath> shows which model was selected

Single-model (GITSEMA_MODEL) usage is fully backward-compatible.

[jsilvanus]

@copilot search queries should use both models and search with both!

Fix merge conflicts.

If we added time based constraints, it should so support tag and commit in addition to time.

[Copilot]

@copilot search queries should use both models and search with both!

Fix merge conflicts....

Done in 3a55300. Three things addressed:

Search uses both models (when configured)
gitsema search now embeds the query with both GITSEMA_TEXT_MODEL and GITSEMA_CODE_MODEL, runs two separate vector searches (each filtered to embeddings produced by the matching model), then merges and deduplicates results keeping the best score per blob. Single-model behaviour is unchanged.

search.ts was inconsistent — it still used GITSEMA_MODEL / single-provider. Updated to use GITSEMA_TEXT_MODEL / GITSEMA_CODE_MODEL to match index.ts.

--since now accepts dates, tags, and commits

gitsema index --since 2024-01-01        # date → git rev-list --after=
gitsema index --since v1.2.0            # tag → v1.2.0..HEAD range
gitsema index --since HEAD~100          # symbolic ref → HEAD~100..HEAD
gitsema index --since abc1234           # commit hash → abc1234..HEAD

[jsilvanus]

@copilot merge conflicts! Fix!

[Copilot]

@copilot merge conflicts! Fix!

Fixed in 1f6e5bf. Merged origin/main (Phases 6 & 7) into this branch and resolved four conflicts:

• blobStore.ts — both FileCategory (Phase 8) and CommitEntry (Phase 6) imports kept
• indexer.ts — Phase 8's since option kept alongside Phase 6's commit-map Phase B walk
• vectorSearch.ts — merged model filter option (Phase 8) with recent/alpha/before/after time options (Phase 7); mergeSearchResults retained
• search.ts — dual-model path now passes all time options (recent, alpha, before, after) through to both vectorSearch calls; orphaned old single-model block removed