Embedding dimension validation is missing in pass-through mode

[devigned]

Problem

KD6 does not enforce embedding dimensionality consistency within a store when running in pass-through mode (KD6_EMBEDDING_PROVIDER=none). This violates OMS spec §8.5.2:

All embeddings within a single store MUST have the same dimensionality. The dimensionality is determined by the store's configured embedding provider (or by the first caller-provided embedding if no provider is configured). Subsequent writes with mismatched dimensionality MUST be rejected with 422 Unprocessable Entity.

How we discovered this

While building the CrewAI integration example, we configured KD6 with the local fastembed embedder (all-MiniLM-L6-v2, 384 dimensions) on the server side, while CrewAI was independently generating embeddings via Azure OpenAI (text-embedding-3-small, 1536 dimensions) on the client side.

Both clients were writing to the same store. The client-supplied 1536-dim vectors were stored alongside 384-dim vectors with no error. Cosine similarity between vectors of different dimensions produces either garbage scores or panics, depending on the implementation. We sidestepped the issue by switching to KD6_EMBEDDING_PROVIDER=none (pure pass-through), but the underlying validation gap remains.

The kagent project solves this more aggressively by enforcing a fixed 768-dimension contract with truncation and L2 normalization, guaranteeing all vectors are compatible regardless of which provider generated them. We do not need to go that far, but we do need to enforce the spec.

Impact

1. Silent data corruption. Mixed-dimension vectors in a store produce meaningless cosine similarity scores. Search results are garbage, but no error is reported — the worst kind of failure mode.

2. Pass-through mode has zero validation. When is_noop(provider) returns true, all dimension checks in auto_embed_content, auto_embed_query, and auto_embed_update are skipped (embedding.rs:129-136). The spec explicitly states that pass-through stores should lock dimensionality based on the first caller-provided embedding.

3. No store-level dimension tracking. The store schema has no column to record the embedding dimensionality. Without this, the server cannot validate subsequent writes even if it wanted to. The max_embedding_dimensions field in ProviderCapabilities is always None (provider.rs:325).

4. Multi-client scenarios are fragile. Any store serving multiple clients (e.g., a CrewAI adapter and a direct curl user, or two different agent frameworks) is vulnerable. If one client switches embedding providers, the store silently degrades.

Current behavior

# Start KD6 in pass-through mode
KD6_EMBEDDING_PROVIDER=none cargo run -p kd6-server

# Write a 3-dim embedding
curl -X POST localhost:8080/v1/stores/test/memories \
  -H "X-Tenant-ID: t1" -H "Content-Type: application/json" \
  -d '{"content": "hello", "embedding": [1.0, 0.0, 0.0], "layer": "semantic", "owner_agent_id": "a"}'
# → 201 Created ✓

# Write a 1536-dim embedding to the SAME store — should be rejected, but isn't
curl -X POST localhost:8080/v1/stores/test/memories \
  -H "X-Tenant-ID: t1" -H "Content-Type: application/json" \
  -d '{"content": "world", "embedding": [0.1, 0.2, ...1536 floats...], "layer": "semantic", "owner_agent_id": "a"}'
# → 201 Created ✗ (should be 422 Unprocessable Entity)

What the spec requires (§8.5.2)

1. The first embedding written to a store (when no provider is configured) locks the store's dimensionality.
2. Subsequent writes with a different dimensionality MUST be rejected with 422 Unprocessable Entity.
3. This applies to both memory creation and search (mismatched query embeddings should also be rejected).

Proposed solution

1. Add embedding_dimensions to the store schema

ALTER TABLE stores ADD COLUMN embedding_dimensions INTEGER;

This is NULL initially and set on the first embedding write (either from the configured provider's dimensions() or from the first caller-provided vector length).

2. Validate on every embedding write

In the auto_embed_content / auto_embed_update helpers (or at the route handler level), check:

• If store.embedding_dimensions is NULL and an embedding is provided, set it to emb.len() (atomic update).
• If store.embedding_dimensions is set and emb.len() != store.embedding_dimensions, return 422.

This must happen even in pass-through mode — the is_noop() guard in current validation code should not skip this check.

3. Validate on search

Query embeddings must also match store.embedding_dimensions. Reject with 422 if they don't.

4. Expose in capabilities

The GET /v1/stores/{store_name} response should include embedding_dimensions so clients can discover the expected dimension before writing.

Spec reflections

The OMS spec (§8.4 and §8.5.2) already covers this well in theory. Two areas could be strengthened:

1. §8.5.2 could be more explicit about pass-through mode. The current text says dimensionality is determined "by the first caller-provided embedding if no provider is configured" — this is correct but easy to miss. A dedicated subsection for pass-through dimension locking would make the requirement more visible to implementors.

2. The spec does not address the semantic compatibility problem. Even when dimensions match, two different models producing 1536-dim vectors will occupy different semantic spaces. The spec acknowledges this in §8.4 ("Callers who provide their own embeddings are responsible for using a model that produces vectors in the same semantic space") but offers no enforcement mechanism. Possible future additions:

   • Store-level embedding_model_id tracking (already mentioned in §8.5.1)
   • Optional warnings when a client-provided embedding's model doesn't match the store's recorded model
   • Optional normalization (like kagent's truncate + L2-normalize to a target dimension)

These are not blockers for the current fix but worth tracking for spec v0.2.