Skip to content

index(action="build", force=true) cannot migrate a SQLite-vec index across embedding dimensions #18

Description

@mic-urs

$## Summary\nindex(action="build", force=true) does not recover from an existing SQLite-vec index whose vector dimension differs from the configured embedder.\n\n## Reproduction\n1. Build a project index with an embedder that returns 768-dimensional vectors.\n2. Configure an embedder that returns 384-dimensional vectors.\n3. Run index(action="build", force=true).\n4. Query index(action="status").\n\n## Actual behavior\nThe tool initially returns status: started, but the background job fails with the original dimension mismatch. The immutable code_vec table remains at 768 dimensions; no replacement index is created.\n\n## Expected behavior\nWith force=true, the tool should make the documented recovery path work: remove/recreate the incompatible project vector table and rebuild it at the configured dimension, while leaving unrelated projects intact. The outcome must be visible in final status.\n\n## Root cause\nRetrievalClient::sync_project calls guard_index_dim before it reaches the force-reindex work in stream_index. guard_index_dim rejects a mismatch unconditionally, so force_reindex cannot affect a dimension migration.\n\nRelevant locations:\n- src/retrieval/sync.rs: sync_project calls guard_index_dim before stream_index;\n- src/retrieval/client.rs: guard_index_dim returns the mismatch error;\n- src/retrieval/sqlite_code_store.rs: the vector table dimension is immutable.\n\n## Regression coverage requested\nUse a temporary SQLite-vec store to build at one dimension, then call the project-level forced rebuild with another. Assert: the old table is replaced, the final dimension matches the new embedder, all current files are searchable, and a non-forced rebuild still rejects the mismatch.\n\n## Impact\nChanging embedding models requires manual database manipulation despite force=true advertising a full reindex.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions