Wire Builtin embedder into write-time vector regeneration

[coreyt]

Background

Phase 12.5 (merged in 82e0654) wired a read-time query embedder via QueryEmbedder + EmbedderChoice { None, Builtin, InProcess }, plus a Candle-based BuiltinBgeSmallEmbedder using BAAI/bge-small-en-v1.5 behind the default-embedder Cargo feature. Callers opting in via EngineOptions::with_embedder(EmbedderChoice::Builtin) now get a working search() vector branch on natural-language queries (strict text → relaxed text → vector fusion, with [CLS] + L2 pooling, lazy model load, graceful degradation).

Write-time vector regeneration still runs through the subprocess path only: regenerate_vector_embeddings_with_policy in crates/fathomdb-engine/src/admin.rs shells out to VectorRegenerationConfig.generator_command for each chunk batch. Callers who want the Builtin embedder on write cannot use it today — they must configure an external subprocess that produces the same BAAI/bge-small-en-v1.5 vectors.

Problem

Asymmetry: a fathomdb built with --features default-embedder can do read-time embedding with zero external config, but write-time regeneration still requires a separate executable. This means:

1. Callers who want the Builtin embedder on both sides have to build and ship a separate subprocess binary that produces the same vectors — defeating the "batteries included" value proposition of the feature.
2. Read-time and write-time vectors can drift if the subprocess binary's model/version/policy doesn't exactly match BAAI/bge-small-en-v1.5 with CLS + L2 — silent retrieval quality regression.
3. The write-time contract (model_identity, model_version, dimension, normalization_policy) is already designed around the Builtin identity (BAAI/bge-small-en-v1.5, version pin, 384, l2) but no code path writes it from the in-process embedder.

Proposed scope

Add a new in-process write-time regeneration path that accepts a &dyn QueryEmbedder (or a dedicated BatchEmbedder sibling trait with batch_embed(&[String]) -> Result<Vec<Vec<f32>>, EmbedderError>) and persists the contract from embedder.identity(). Shape:

• regenerate_vector_embeddings_in_process(embedder: &dyn BatchEmbedder, ...) alongside the existing regenerate_vector_embeddings_with_policy.
• BuiltinBgeSmallEmbedder grows a batch_embed method that runs the BERT forward pass over a batch in one tokenizer call (should be a modest efficiency win over looping single-query calls).
• Admin surface (Rust, Python, TS) gains a variant / option to choose "in-process builtin" over "subprocess" for regeneration. Preserve the subprocess path for BYO callers.
• Contract persistence reuses the existing model_identity / model_version / dimension / normalization_policy / chunking_policy / preprocessing_policy row shape — no schema change.
• Tests: feature-gated integration test that registers a schema, writes nodes, triggers in-process regeneration, and verifies the contract row matches the embedder identity and the emitted vectors cosine-match the query-time vectors for the same text.

Non-goals

• No change to read-time embedder plumbing — Phase 12.5 ships that.
• No new embedder trait that duplicates QueryEmbedder. A BatchEmbedder: QueryEmbedder sibling is fine if it cuts tokenizer overhead.
• No auto-migration of existing databases — callers trigger regeneration explicitly as today.
• No removal of the subprocess generator_command path. BYO callers keep it.

Sequencing

Natural follow-up after Phase 12.5 merges. No blockers from other phases.

References

• Phase 12.5 rollout: commits 1a17f10..82e0654
• Read-time embedder design: dev/design-adaptive-text-search-surface-addendum-1-vec.md (see "v1.5 update" section)
• Existing write-time path: crates/fathomdb-engine/src/admin.rs:265-278 (VectorRegenerationConfig), admin.rs:3493 (validate_generator_command), admin.rs:2794 (regenerate_vector_embeddings_with_policy)
• Read-time trait: crates/fathomdb-engine/src/embedder/mod.rs, crates/fathomdb-engine/src/embedder/builtin.rs