Summary
Add runtime-configurable embedding backend selection and incremental per-symbol re-embedding to Brain Mode.
Currently, the embedding backend (hashing-trick 256d vs nomic pretrained 768d) is selected at compile time via #[cfg(feature = "pretrained-embed")]. This means users cannot switch backends without recompiling. Additionally, embed_project() re-embeds all symbols in a project on every index run, even when only 1 file changed.
Problem
- No runtime config — embedding backend is compile-time only, no
.cora.yaml control
- Full re-embed on every index —
embed_project() queries SELECT ... FROM symbols WHERE project_id = ? with no changed-symbol filter
- No
brain section in .cora.yaml config schema
Proposed Changes
1. BrainConfig in .cora.yaml
brain:
embedding: auto # auto | hashing | pretrained
| Value |
Behavior |
auto (default) |
Best available: pretrained (if compiled) → hashing fallback |
hashing |
Force 256d hashing trick (zero dependency) |
pretrained |
Force nomic 768d (requires --features pretrained-embed) |
New struct in src/config/schema.rs (~15 lines).
2. Runtime Embedding Dispatch
Refactor src/embed/mod.rs from compile-time #[cfg] to runtime enum dispatch:
pub enum Backend {
Hashing, // always available
Pretrained, // cfg(feature = "pretrained-embed")
}
pub fn resolve_backend(config: &BrainConfig) -> Backend { ... }
The embed_code_dispatch() function reads the resolved backend instead of compile-time cfg.
3. Incremental Per-Symbol Embedding
Migration v7: Add embed_fingerprint TEXT column to symbols table.
ALTER TABLE symbols ADD COLUMN embed_fingerprint TEXT;
brain.rs: embed_project() only embeds symbols whose name + signature hash differs from stored fingerprint:
// Before: embed ALL symbols
SELECT id, name, kind, signature FROM symbols WHERE project_id = ?1
// After: embed ONLY changed symbols
SELECT id, name, kind, signature FROM symbols
WHERE project_id = ?1 AND (
embed_fingerprint IS NULL
OR embed_fingerprint != ?
)
Performance Impact
| Scenario |
Before (full re-embed) |
After (incremental) |
| Full index (cold, 1106 symbols) |
0.9s |
0.9s (same — all new) |
| 1 file changed (~89 symbols) |
0.6s |
~0.05s |
cora watch auto-reindex |
0.6s per change |
~0.05s per change |
Benchmarked on Uteke (1106 symbols, 147 files) with nomic 768d backend.
Files to Change
| File |
Change |
Est. Lines |
src/config/schema.rs |
Add BrainConfig struct |
+15 |
src/config/loader.rs |
Add brain defaults |
+5 |
src/embed/mod.rs |
Runtime dispatch + Backend enum |
+50 (refactor) |
src/index/schema.rs |
Migration v7: embed_fingerprint column |
+15 |
src/index/brain.rs |
Incremental embed in embed_project() |
+30 (refactor) |
src/index/mod.rs |
Pass brain config to embed call |
+3 |
Total: ~120 lines changed across 6 files.
Testing
Related
Summary
Add runtime-configurable embedding backend selection and incremental per-symbol re-embedding to Brain Mode.
Currently, the embedding backend (hashing-trick 256d vs nomic pretrained 768d) is selected at compile time via
#[cfg(feature = "pretrained-embed")]. This means users cannot switch backends without recompiling. Additionally,embed_project()re-embeds all symbols in a project on every index run, even when only 1 file changed.Problem
.cora.yamlcontrolembed_project()queriesSELECT ... FROM symbols WHERE project_id = ?with no changed-symbol filterbrainsection in.cora.yamlconfig schemaProposed Changes
1. BrainConfig in
.cora.yamlauto(default)hashingpretrained--features pretrained-embed)New struct in
src/config/schema.rs(~15 lines).2. Runtime Embedding Dispatch
Refactor
src/embed/mod.rsfrom compile-time#[cfg]to runtime enum dispatch:The
embed_code_dispatch()function reads the resolved backend instead of compile-time cfg.3. Incremental Per-Symbol Embedding
Migration v7: Add
embed_fingerprint TEXTcolumn tosymbolstable.brain.rs:
embed_project()only embeds symbols whosename + signaturehash differs from stored fingerprint:Performance Impact
cora watchauto-reindexBenchmarked on Uteke (1106 symbols, 147 files) with nomic 768d backend.
Files to Change
src/config/schema.rsBrainConfigstructsrc/config/loader.rssrc/embed/mod.rsBackendenumsrc/index/schema.rsembed_fingerprintcolumnsrc/index/brain.rsembed_project()src/index/mod.rsTotal: ~120 lines changed across 6 files.
Testing
cargo test --features tree-sitter— all 903+ tests passcargo test --features tree-sitter,pretrained-embed— pretrained path testedcargo clippy --all-targets --features tree-sitter -- -D warningscargo clippy --all-targets --features tree-sitter,pretrained-embed -- -D warningsvectorsignal.cora.yamlwithbrain: { embedding: hashing }forces 256d.cora.yamlwithbrain: { embedding: pretrained }forces 768dautodefault worksRelated
embed_projecton file change)