mcp-data-platform-v1.74.0

Overview

This release makes memory recall robust to embedder outages and self-healing across model swaps. Two changes ship together:

1. Hybrid recall + lexical fallback - memory_recall now fuses vector similarity with a Postgres full-text match in a single ranked query, and degrades to lexical-only (rather than erroring) when no embedding provider is available.
2. Memory as an index-jobs consumer - memory joins the shared index-jobs framework (source_kind = "memory"), so records left without a vector (saved during an outage) or stamped with a stale model self-heal off the request path. Memory now also appears on the admin Indexing dashboard added in v1.73.0.

Migration 000054 adds two indexes and two breadcrumb columns to memory_records. There is one new package (pkg/memory/memoryindex) and one new recall strategy (lexical). No configuration change is required, and no existing tool changes its inputs.

Why

Before this release, memory_recall's semantic path embedded the query and ran an ORDER BY embedding <=> $1 cosine scan with no index - an O(n) sequential scan over memory_records - and treated a missing or failed embedder as a hard error. Two consequences:

• An embedder outage broke recall entirely. A query that could not be embedded returned an error, even though the rows it wanted were sitting in Postgres and matchable by keyword.
• Outage-saved and model-swapped rows stayed stale forever. A memory saved while the embedder was down had a NULL embedding and was invisible to vector search; nothing brought it back. A provider/model swap left every prior row indexed under the old model with no path to re-embed.

The synchronous embed-on-write path remains (read-your-writes), but it cannot cover the rows it never got to write. This release closes both gaps.

What it does

Hybrid recall

memory_recall's semantic and auto strategies now run hybrid retrieval: a two-arm UNION ALL over a vector arm (cosine via the new HNSW index) and a lexical arm (Postgres full-text via the new GIN index), with the two signals fused in Go into one rank score:

score = 0.6 * semantic + 0.4 * lexical

semantic is the query/record cosine mapped to [0,1]; lexical is a binary "the content matched the full-text query" signal. Leaning semantic (0.6) keeps free-form intent queries ranking by meaning, while the lexical term gives an exact-identifier hit (entity URN, column name, error code) a decisive boost over a merely semantically-near row. The weight and the binary-lexical blend deliberately match pkg/toolkits/apigateway/ranking.go, so both toolkits rank hybrid results on the same curve.

The response now reports a ranking field (hybrid, lexical, entity, or graph) so the caller knows which path produced the results.

Lexical fallback (graceful degradation)

When the embedder errors or returns a zero vector (the noop placeholder), recall no longer fails. It falls back to lexical-only retrieval and flags the response:

{ "ranking": "lexical", "degraded": true,
  "note": "embedding provider unavailable; results are lexical-only (exact-term matches), not semantic" }

The lexical arm also surfaces NULL-embedding rows that vector search skips - exactly the rows saved during an outage. A new explicit lexical strategy is the forced counterpart to this automatic fallback (useful for exact-token lookups); it is not flagged degraded because lexical was the caller's choice. In auto, a semantic-arm degradation propagates so the lexical-only signal is visible even through the merged result.

Memory as an index-jobs consumer (backfill)

Memory registers on the shared index-jobs queue (pkg/indexjobs) as source_kind = "memory", joining the api-catalog and tools consumers. A unit is one record; the worker embeds its content and writes the vector back onto the same row. Gap detection is condition-based: FindGaps returns active records whose embedding is NULL or whose embedding_model differs from the current provider, so:

• records saved while the embedder was down get embedded once it recovers, and
• a model swap re-embeds every row stamped with the previous model.

The synchronous embed-on-write path is preserved and now stamps embedding_model and embedding_text_hash, so a healthy row never looks like a gap and unchanged content is not re-embedded. Memory now appears on the admin Indexing dashboard (v1.73.0) via the shared coverage reporter, with ExpectedKnown = true (every active record is expected to converge to one vector).

The memory consumer registers only when the memory store is wired, and the whole queue stands up only when the platform has both a database and a configured (non-noop) embedding provider. File-mode and no-embedding deployments fall back to lexical ranking with no queue.

Migration

000054_memory_hybrid_search alters the existing memory_records table (no new tables):

• idx_memory_records_embedding_hnsw - an HNSW ANN index on embedding using vector_cosine_ops, replacing the prior O(n) cosine scan. Requires pgvector >= 0.5.0 (the platform's pgvector/pgvector:pg16 image ships it).
• idx_memory_records_content_fts - a GIN index on to_tsvector('english', content) backing the lexical arm.
• embedding_model TEXT NOT NULL DEFAULT '' and embedding_text_hash BYTEA - the provider-identity and content-hash breadcrumbs index-jobs uses to dedup re-embeds and detect model-swap gaps. Embedding dimension is not stored; it is derivable from the stored vector's length.

The migration re-enables the vector extension defensively so it is self-contained.

How it works

• HybridSearch (pkg/memory/postgres.go) issues the UNION ALL so both the HNSW and GIN indexes are used, scans candidates with collectHybridRows, and fuses/dedups/orders them in Go via rankFused (using fuseHybridScore). clampStoreLimit bounds every search path's limit to [1, MaxLimit].
• recallBySemantic embeds the query and chooses hybrid vs. lexical fallback; isZeroVector detects the noop provider's all-zero output. lexicalOutcome centralizes the active-vs-stale status policy across the semantic, lexical, and auto paths.
• pkg/memory/memoryindex implements Source, Sink, and Store for the memory kind. Sink also implements CoverageReporter; StampExpected is a no-op because gap detection is condition-based, not count-based. embedding.ModelName is now exported and supplies the current model to the Sink (indexjobs.providerModel delegates to it).
• Wiring lives in Platform.registerIndexConsumers (pkg/platform/apigateway_embed_jobs.go): the memory consumer is registered alongside api-catalog and tools, and the reconciler's FindGaps discovers stale rows directly from memory_records, so no startup bootstrap enqueue is needed.

Upgrade notes

• Run migrations as usual; 000054 is additive (two indexes, two nullable/defaulted columns) and self-contained.
• No configuration change is required. On a deployment with a database and a configured embedding provider, existing rows backfill automatically as the index-jobs reconciler finds them; recall serves hybrid results immediately for already-embedded rows.
• On deployments without an embedding provider (file-mode or no-embedding), recall serves lexical-only results and reports degraded/ranking: lexical; no queue is started.

Compatibility

• memory_recall inputs are unchanged. The response adds ranking, and adds degraded/note only on the lexical-fallback path. A new optional lexical strategy value is accepted.
• No existing tool, admin endpoint, or config key changes behavior.
• The api-catalog and tools embedding consumers are behaviorally unchanged; memory is an additional consumer on the same queue.

Changelog

Features

• 83f8403: feat(memory): hybrid recall with lexical fallback + index_jobs backfill (#507) (#508) (@cjimti)

Installation

Homebrew (macOS)

brew install txn2/tap/mcp-data-platform

Claude Code CLI

claude mcp add mcp-data-platform -- mcp-data-platform

Docker

docker pull ghcr.io/txn2/mcp-data-platform:v1.74.0

Verification

All release artifacts are signed with Cosign. Verify with:

cosign verify-blob --bundle mcp-data-platform_1.74.0_linux_amd64.tar.gz.sigstore.json \
  mcp-data-platform_1.74.0_linux_amd64.tar.gz