Skip to content

feat(phase-60): vector and topic quality fixtures (60-03) - #50

Open
RichardHightower wants to merge 1 commit into
feature/phase-60-live-backend-isolationfrom
feature/phase-60-quality-fixtures
Open

feat(phase-60): vector and topic quality fixtures (60-03)#50
RichardHightower wants to merge 1 commit into
feature/phase-60-live-backend-isolationfrom
feature/phase-60-quality-fixtures

Conversation

@RichardHightower

Copy link
Copy Markdown
Contributor

Summary

Phase 60-03 of v3.2 Prove It (#46). Stacked on #49 (60-01).

A paraphrase set BM25 cannot solve, a --layers bm25|vector|hybrid switch on the custom harness, and a labelled TopicExtractor::cluster artifact behind the README status table.

Honest caveats (do not skip these):

  • Mock BM25 is token overlap.
  • Mock vector is a committed paraphrase lexicon + TF-IDF cosine, not Candle/HNSW.
  • Mock hybrid is RRF (k=60) of those two lists, not live RouteQuery.
  • --layers is mock-only; CLI memory search is always hybrid.
  • memory-bench all excludes semantic so custom-harness-mock.json is not tanked.
  • Topic purity/ARI is a synthetic 80-doc TF-IDF corpus, not live TOC / Candle embeddings.

Committed numbers

Artifact layers recall@5 pass
benchmarks/results/semantic-bm25.json bm25 0.00 0/16
benchmarks/results/semantic-vector.json vector 1.00 16/16
benchmarks/results/semantic-hybrid.json hybrid 1.00 16/16
benchmarks/results/topics-quality.json purity 1.00 / ARI 1.00 80 docs, 8 clusters, 0 noise

Closes BENCH-13 / QUAL-01 / QUAL-02 / QUAL-03. Evidence for #40 and #47.

Does not start 60-02 (#39 — maintainer-owned).

Acceptance

  • ≥15 semantic tests; hit files share no query tokens (semantic_hits_do_not_contain_query_tokens)
  • BM25 recall@5 < 0.4 on the paraphrase set (committed 0.00)
  • hybrid recall@5 > bm25 recall@5 (1.00 > 0.00)
  • --layers on semantic / run / all
  • topics-quality.json from TopicExtractor::cluster
  • cargo test -p memory-topics --lib includes hand-computed purity 8/9 and ARI 4.5/7
  • README vector/topic rows cite the artifacts; positioning Claims Ledger added

Test plan

  • cargo test -p memory-bench --lib (55 passed, including QUAL-01 leak + BM25<0.4 tests)
  • cargo test -p memory-topics --lib (167 passed, including metrics + quality)
  • cargo clippy -p memory-bench --all-targets -- -D warnings
  • cargo clippy -p memory-topics --lib -- -D warnings
  • rustfmt --check on the touched crates

Full task pr-precheck is workspace-wide; same targeted bar as #49.

Paraphrase set BM25 cannot solve, --layers bm25|vector|hybrid on the
custom harness, and a labelled TopicExtractor clustering artifact
behind the README status table.

Mock BM25 is token overlap. Mock vector is a committed lexicon plus
TF-IDF cosine, not Candle. Mock hybrid is RRF k=60. memory-bench all
excludes semantic so the original custom-harness number is not tanked.

Committed: BM25 recall@5 = 0.00 (0/16), vector 1.00 (16/16), hybrid
1.00 (16/16); topics purity 1.00 / ARI 1.00 on synthetic TF-IDF.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants