Round 5: semantic grouping, card-aware ranking, reranker stage#273
Merged
Conversation
New module vnext_reranker: provider-side listwise relevance scoring of the fused candidate head (up to 48 memories + 24 sources per pack) via any OpenAI-compatible /chat/completions endpoint, configured with ALICE_RERANKER_BASE_URL / ALICE_RERANKER_MODEL / ALICE_RERANKER_API_KEY (the vnext_embeddings env pattern). Unconfigured, the stage is dormant: zero provider calls, fused order stands, packs byte-identical (tested). Semantics: reranking reorders, never shrinks — the same number of selection slots is filled afterwards and the token-budget packer still decides what survives; policy-excluded candidates are never re-admitted; equal scores fall through the round-3 content-stable cascade (imported, not duplicated); any provider failure fails open to fused order with the failure recorded. minimal depth skips the stage with an honest status. Honesty: the scoring prompt is a frozen generic relevance prompt (no query-type or benchmark vocabulary), committed as a constant and sha-pinned in tests; every configured compile discloses the stage in trace.stages.reranker with provider, model, prompt sha, candidates scored, reorder flag, latency, and token usage. The free coverage probe scrubs ALICE_RERANKER_* by default (keyless-determinism promise) and gains --with-reranker for the provider-backed phase-2 probe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ings
Round-4 measured the lexical ceiling on aggregation recall: for questions
whose evidence shares NO anchor token ("How many kitchen items did I
replace or fix?" = faucet/toaster/shelves, ~8 of 15 sampled aggregation
misses), no entity or lexical-topic group can ever form. This adds a third
grouping pass to vnext_rollups that runs AFTER the lexical/entity passes
claim their members, over only the unclaimed remainder:
- Embedding access reuses the consolidation pass's exact pattern
(provider re-embed from memory_embedding_text plus ONE
search_memories_vector probe as the stored-embeddings read surface);
rows without a stored vector never participate.
- Unclaimed rows agglomerate by pairwise cosine (single-linkage connected
components) at one threshold chosen from a conservative sweep
(0.60-0.85) by a silhouette-style internal criterion — deterministic
given fixed vectors; ties break toward the higher threshold.
- Every semantic cluster passes the SAME round-4 utility gate (>= 3
members, aggregation signal) with one documented substitution: group
coherence is a mean-pairwise-similarity floor (0.60) instead of the
label-stem majority test, which anchor-less groups cannot pass by
construction. Clusters at near-duplicate similarity (>= 0.88) are left
to the dedup/merge pipeline.
- The card label is the dominant noun across member texts (member
support, occurrences, alphabetical; verbs/closed-class/store-generic
stems never label; must span >= 2 members) and runs the full round-4
structural hygiene; titles keep the unit-bearing round-4 rendering.
- NO embeddings -> the tier is dormant: outcome metadata, skip lines, and
candidates are byte-identical to the lexical/entity-only behavior
(guarded by tests and verified byte-for-byte across 12 real
LongMemEval stores against main). When the tier runs, the sweep
scores, chosen threshold, cluster counts, and per-reason skips are all
disclosed in outcome.semantic / rollups.semantic_grouping and the
consolidation report.
Wiring: VNextRollupService gains an embedding_provider seam (ambient env
resolution identical to the consolidation service); the consolidation
workflow passes its already-resolved provider through so both passes see
one seam. No new options, tables, or commit-path work: the tier rides
the scheduled consolidation workflow only, and the O(1) commit-path
statement-identity guard still passes.
Tests: mock-vector fixtures prove the anchor-less kitchen trio forms
exactly one gated topical card while unrelated rows stay out; gate
enforcement on semantic groups (no aggregation signal, similarity-floor
chains, no dominant noun, near-duplicate hand-off); dormancy byte
identity; determinism across insertion order and reruns; consolidation
workflow integration; and a live-SQLite propose -> review-accept ->
aggregation-recall loop where the accepted card reaches the context pack
for the round-4 question.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion queries Round-4 measured the inversion: topical accepted roll-up cards ranked 17th/20th/34th while their own member memories ate the selection slots (4/15 aggregation stores packed a card at 16 slots). The card IS the aggregate answer — its instance list pre-aggregates the members — and the members are its receipts, yet RRF ranks the card below them because every member matches the query about as well and there are more of them. New coverage-mode pass (vnext_coverage_query.promote_rollup_cards), composed into compile_context_pack in a marked block AFTER the instance diversity pass, gated by the existing detect_aggregation_intent surface gate only: - Accepted cards are recognized by the exact acceptance shape (metadata_json.consolidation with proposal_kind="rollup", cluster_member_ids, and the accepted stamp accept_consolidation_candidate writes); unaccepted or merge/dedup candidates never promote. The proposal-kind literal is pinned to vnext_rollups.ROLLUP_PROPOSAL_KIND by a unit test. - A card promotes only when the inversion is real: at least COVERAGE_MIN_SLOTTED_MEMBERS (2) of its members hold selection slots and the best one outranks the card. Measured on the free probe, a lone slotted member is an ordinary hit — promoting on one never surfaced an evidence-bearing card and once displaced the sole carrier of an evidence session. - The card takes exactly the best member's rank; members stay in the pool directly below it as receipts (demote-not-drop — only the last slot holder loses selection, honestly trimmed in the trace). - At most COVERAGE_MAX_CARD_PROMOTIONS (2) cards promote per pack, so a query grazing several roll-up topics is not flooded by cards. - Disclosed as card_promotions on the coverage_mode trace stage record. - Dormant byte-identity: no aggregation intent, no accepted cards, no slotted-member plural, or card already first -> the candidate list is returned untouched (same objects) and the pack is byte-identical; policy-excluded candidates never move and are never re-admitted. Free measurement (fresh --accept-rollups ingest of the 15-store round-4 aggregation probe, paired baseline/after on identical stores, keyless): slot wins 4/15 -> 9/15 at 16 slots (1/15 -> 5/15 at the probe-default 8); every in-pack card evidence-bearing (cards covering any evidence 4 -> 9, all evidence 3 -> 5 of 15); zero coverage regressions (any/all and per-question n_hit unchanged at both depths); the four previously winning stores re-render with the card above every member (ranks 6->4, 12->2, 14->1, 11->3) and 5-9 member receipts retained below. Full unit suite (2065), LongMemEval harness tests, and the vNext eval battery pass; integration failures (Postgres-dependent) are identical with the change stashed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The coverage-side builds of the 90% campaign, each free-gated before this PR. No benchmark claims change; the paid measurement phase follows separately.
Features
ALICE_RERANKER_BASE_URL/_MODEL/_API_KEY); reorders, never shrinks; fail-open; generic scoring prompt sha-pinned against drift; dormant unconfigured (byte-identity tested). Active-path quality is deliberately unmeasured here — it requires the phase-2 embeddings-on probe.Verification
2190 unit+harness tests green, six-suite eval battery passes, official templates sha-pinned, honesty audit clean (all triggers query-surface/content-shape; no benchmark-label reads), commit path O(1) guard passes, review-gate semantics untouched.
Upgrade Overview
Protected Areas
Compatibility Impact
No protected paths touched. New optional env (
ALICE_RERANKER_*); retrieval traces gaincard_promotionsand (when configured) arerankerstage record; consolidation outcomes gain asemanticdisclosure block when embeddings are active. All new behavior is gated and dormant by default.Migration / Rollout
No migrations. Normal rollout.
Operator Action
None. Optional: configure
ALICE_RERANKER_*to enable precision reranking; embeddings-on deployments get semantic grouping automatically in the scheduled consolidation pass.Validation
Suites above; 15-store aggregation probe (paired arms, slot wins 4→9, coverage flat); dormancy byte-identity tests for all three features.
Rollback
Revert the merge; all changes additive and gated, no data or schema impact.
🤖 Generated with Claude Code