Skip to content

feat(v3.0): Phase 51 Retrieval Orchestrator#28

Merged
RichardHightower merged 13 commits into
mainfrom
feature/v3.0-phase-51-orchestrator
Apr 28, 2026
Merged

feat(v3.0): Phase 51 Retrieval Orchestrator#28
RichardHightower merged 13 commits into
mainfrom
feature/v3.0-phase-51-orchestrator

Conversation

@RichardHightower
Copy link
Copy Markdown
Contributor

Summary

Lands Phase 51 (Retrieval Orchestrator) for v3.0. Cherry-picked from the local gsd/phase-51-retrieval-orchestrator branch, which executed this work on 2026-03-22 but was never pushed.

Adds a new memory-orchestrator crate that wraps RetrievalExecutor with:

  • Heuristic query expansion (6 unit tests)
  • RRF fusion across multiple indexes with deduplication and consensus boosting
  • Reranker trait + HeuristicReranker + CrossEncoder stub
  • ContextBuilder for MemoryContext assembly with token estimation
  • MemoryOrchestrator wiring the full pipeline + mock LLM reranker integration tests

Phase 51 plans (all complete)

Plan Summary
51-01 Scaffold crate, core types, query expansion
51-02 RRF fusion, reranker trait, context builder
51-03 Wire MemoryOrchestrator + full QA

Origin and provenance

These commits were originally executed on 2026-03-22 on the local-only gsd/phase-51-retrieval-orchestrator branch. They are being landed now as part of reconciling the v3.0 local stack with origin/main. The 12 implementation commits were cherry-picked verbatim; one reconciliation commit on top updates STATE.md and ROADMAP.md to mark Phase 51 complete while preserving the Phase 51.5 entry that landed via PR #27.

Decisions baked into the implementation

  • RerankMode defaults to Heuristic (no LLM cost by default)
  • RankedResult uses f64 for fusion precision; SearchResult uses f32
  • RRF deduplicates by doc_id, keeping first-seen SearchResult
  • HeuristicReranker trims to top 10 (MAX_RESULTS const)
  • Token estimation: chars * 0.75 + 50 overhead
  • MemoryOrchestrator accepts Box<dyn Reranker> via with_reranker() for test injection

Tests

task pr-precheck passes: format, clippy (-D warnings workspace-wide), full test suite, doc build.

Test plan

  • task pr-precheck (already done locally — repeat in CI)
  • Spot-check memory-orchestrator unit tests run: cargo test -p memory-orchestrator
  • Confirm no behavioral changes to memory-retrieval (orchestrator wraps, doesn't modify)
  • Verify Phase 51.5 (PR feat(v3.0): wire API-based summarizer from config (supersedes #26) #27) features still work — cargo test -p memory-summarizer if present

🤖 Generated with Claude Code

RichardHightower and others added 13 commits April 28, 2026 00:19
- Add memory-orchestrator to workspace members and dependencies
- Define OrchestratorConfig, RankedResult, MemoryContext, RerankMode types
- Create stub modules for fusion, rerank, context_builder, orchestrator
- Add 4 unit tests for type defaults and result ordering

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- expand_query generates lowercase and keyword-stripped variants
- Strips leading question words (what, how, why, when, where, did we, do we)
- Always includes original query; handles empty input
- 6 unit tests covering all expansion behaviors

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- SUMMARY.md with 2 task commits and 10 tests documented
- STATE.md advanced to plan 2 of 3, decisions recorded
- ROADMAP.md updated with phase 51 progress
- ORCH-01 and ORCH-07 requirements marked complete

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…osting

- rrf_fuse merges ranked lists using 1/(k+rank) scoring
- FusedResult type wraps SearchResult with cumulative RRF score
- Deduplicates same doc_id across and within lists
- Handles empty input lists gracefully
- 4 tests: order preservation, consensus boost, empty lists, dedup

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…coder stub

- Reranker async trait with query + FusedResult inputs
- HeuristicReranker sorts by RRF score, trims to top 10
- CrossEncoderReranker stub falls back to heuristic with warning
- RerankedResult type for context builder consumption
- 2 tests: order+trim verification, cross-encoder fallback

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Converts RerankedResults to MemoryContext with summary and events
- Token estimation: chars * 0.75 + 50 overhead
- Confidence from top result score, 0.0 for empty
- Summary: "No relevant memory found." or "Found N relevant..."
- 3 tests: empty results, confidence, token estimation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fixes clippy unused-import warning in non-test code

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- SUMMARY.md with 3 tasks, 9 tests, zero clippy warnings
- STATE.md advanced to plan 3 of 3
- ROADMAP.md updated with plan progress
- Requirements ORCH-02 through ORCH-06 marked complete

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…reranker tests

- Implement MemoryOrchestrator<E> with expand -> fan-out -> RRF -> rerank -> context pipeline
- Add with_reranker() constructor for Box<dyn Reranker> injection
- Add MockLlmReranker (cfg(test)) that reverses RRF order for deterministic testing
- Test: fused results with RRF consensus (doc in 2 lists ranks highest)
- Test: fail-open when one layer fails
- Test: LLM rerank reorders results (ORCH-04 satisfied)
- Test: query expansion path executes without error

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix useless vec! in types.rs test (use array instead)
- Fix formatting in rerank.rs test

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reconcile planning files after cherry-picking Phase 51 implementation
from gsd/phase-51-retrieval-orchestrator. The cherry-picks took 'ours'
for STATE/ROADMAP at conflict points (preserving Phase 51.5 metadata
that didn't exist on the local branch); this commit catches them up:

- ROADMAP.md: mark Phase 51 + all 3 plans complete; preserve 51.5 entry
- STATE.md: bump completed_phases 1 -> 2, completed_plans 1 -> 4,
  percent 25 -> 50; add 6 [Phase 51] decisions from local branch;
  update Current Position and Cumulative Stats

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant