Remembra 5.7.0 — Advanced Retrieval Engine (§31–§38)

Roadmap §31–§38 — the Advanced Retrieval Engine milestone.

Full compatibility statement: docs/v5.7.0-compatibility.md.
Benchmark gate: docs/benchmark-gate.md. Audit: docs/v5.7.0-audit.md.

Two default-on behaviour changes

Everything else here is additive with unchanged defaults. These two are not.

1. Exact duplicate results are collapsed. Two records of the same statement are
   one answer to a question. Escape hatch: dedupeExact: false.
   Same-source collapsing is off by default (dedupeSameSource) — identical text
   from two different sources is often corroboration, and discarding the second
   source is a worse error than showing a repeat.
2. Superseded memories are excluded. A query for the current policy no longer
   returns v1 and v2 side by side. Escape hatch: includeSuperseded: true.
   Suppression applies only when the replacement is actually available, so a
   dangling supersededBy keeps the old copy visible rather than replacing it with
   nothing.

Neither deletes anything. The store still holds every version, each readable by id.

New in this release

• Token efficiency and duplicate rate in the evaluation harness — the two of
  §38's seven metrics that were missing, and the guardrail the deduplication work
  needed.
• Whole-token matching, phrase scoring and field weights. A query for cat
  scored "concatenate the streams" at exactly the same 60.00 as "the cat sat",
  because matching was a single String.includes.
• Real inverse document frequency. The old "idf" was computed from the document
  being scored — a coverage discount wearing the name of IDF, which cannot tell a
  rare term from a common one.
• One retrieval budget object covering all four §37 dimensions. Exceeding
  maxLatencyMs returns the best results so far and reports the truncation rather
  than erroring: a bound that returns nothing has not degraded, it has failed.
• Configurable fusion weights via retrieval.fusionWeights or
  REMEMBRA_RETRIEVAL_FUSION_WEIGHTS. Every default reproduces the previous
  behaviour exactly.
• A retrieval benchmark gate is now a release stage. §38 asked that every
  retrieval change be measured; that was a rule people could forget. A 23-scenario
  set now runs through the real pipeline and fails on a regression in precision,
  recall, MRR, nDCG, duplicate rate or a latency percentile.

Fixed

• §37's budget was unreachable and its report was discarded.
  MemoryService.search neither accepted a budget nor returned the report — the
  budget was computed and then dropped, so no caller could set a bound or learn
  which one bound. Both fixed, and the budget is settable over HTTP as
  ?budget.maxItems= and friends, with a malformed bound rejected rather than
  silently ignored.
• The relation-expansion re-rank built its search policy separately from the
  first search, so a configured fusion weight could apply to one ranking and
  silently not the other.

Known limitations

• Non-ASCII queries return no lexical results. extractQuery filters query
  terms with /^[a-z0-9]+$/u, so a query in any non-ASCII script produces zero
  terms and retrieval falls back to the vector path alone. The CJK branch in
  tokenize is therefore unreachable from the query path. Documents in those
  scripts are stored and tokenised correctly; querying is what is affected. Not
  fixed here because the one-line change alters what existing deployments get back
  for non-ASCII input. Full analysis in docs/v5.7.0-audit.md (S7).

Unchanged contracts

Four runtime dependencies, unchanged: @modelcontextprotocol/sdk,
better-sqlite3, yaml, zod. redis remains an optional peer and is not even a
devDependency. No new subpath, no removed export, /health/ready byte-identical to
5.6.0, and every V4.9/V5.0.3 security boundary intact.

Gates

923 test suite · 142 security matrix · 52 recovery matrix · Python SDK ·
47 documentation files · 23 benchmark scenarios · 0 audit findings · 0 npm
vulnerabilities. Full release:check passed for @hilbras/remembra@5.7.0, on
Node 18.20.8 and Node 24.21.0.