0.6.0 — evidence-bound retrieval
First release with a published head-to-head against a comparator on an
externally authored corpus, and the first where a measured result caused work
to be deleted rather than shipped.
Against self-hosted Mem0: parity, not superiority
Mr.TyDi Indonesian, 25 queries over 100 documents, 10 repeats, shared embedding
endpoint. mem0ai 2.0.13 in library mode.
| Titen 0.6.0 | Titen 0.5.7 | Mem0 | |
|---|---|---|---|
| recall@1 | 0.8000 | 0.6200 [0.52–0.68] | 0.7200 |
| MRR@10 | 0.8667 | 0.7767 | 0.8180 |
| nDCG@10 | 0.9000 | 0.8336 | 0.8632 |
| query p50 | 188 ms | 188 ms | 361 ms |
| ingest | 1.8 s | 1.9 s | 17.5 s |
Point estimates favour Titen on every metric. That is not a win. Both systems
are deterministic on this harness, so disjoint across-repeat ranges prove nothing;
the uncertainty that governs a between-system claim is query sampling, and paired
sign tests give p ≥ 0.500 on every metric. The whole margin is two queries —
20 of 25 rank-1 hits against 18 — and Titen loses one query to both Mem0
configurations.
What the run does support: retrieval parity, ~1.9× lower query latency
(~400× without vector search), ~9.6× faster ingest, and deterministic ranking
when vector search is enabled. Full ruling on permitted and forbidden claims:
neutral head-to-head.
What was deleted
An adversarial review rejected the first retrieval change. Isolated at 10
repeats, the score-formula rewrite earned nothing the tie-break alone does not,
regressed the FTS-only lane, and coupled the lexical/semantic balance to one
provider's cosine distribution through a fitted constant. It was removed.
#227 stays open; #226 is narrowed, not closed. What shipped is four lines in
the comparator.
Correction to the record
Figures published earlier the same day for 0.5.7 — recall@1 0.680, MRR@10 0.807,
p50 170 ms — were the top of their ranges over five repeats, not medians. The
true 0.5.7 median is 0.62. Do not requote them.
Newly measured
- One core, every concurrency level. A single client saturates the service at
10k claims (#259). - Recall falls with corpus size: 1.00 → 0.81 → 0.49 across 10³ → 10⁵ (#260).
- Data-usable upgrade floor is 0.2.0. A 0.1.x store migrates without error
and then answers 404 on every claim (#257, #258). - Cold start stays ~53 ms and resident memory grows 2.9% across a hundredfold
corpus.
Added
top_k on context compile, a standard-library-only Python client, a container
image, and the release-bound retrieval harness the evaluation spec had been
waiting for.