feat(genome): demand-aligned-recall PR-3b — LocalDemandAlignedRecall ranking engine by joelteply · Pull Request #1372 · CambrianTech/continuum

joelteply · 2026-05-17T03:34:43Z

Summary

PR-3b of demand-aligned-recall (GENOME-FOUNDRY-SENTINEL Part 7). Composes PR-3a's scoring function (#1371) with a candidate-injection API to produce ranked RankedPools.

PR-3c adds the working-set walker that sources candidates from the substrate (#1362's bus hook); PR-3b stays pure ranking — reviewable as spec compliance + sorting correctness.

What lands

CandidateArtifact — caller-provided candidate ready for scoring. Carries per-factor inputs (semantic, outcome, provenance) + residency + last-used timestamp.
LocalDemandAlignedRecall { weights, half_life_ms } — the ranking engine. Thread-safe through immutability.
new() / with_config(weights, half_life_ms) constructors.
rank(now_ms, candidates) -> RankedPool — pure-function ranking:
- Scores each via PR-3a's score()
- Partitions by PageKind into layers / experts / engrams
- Sorts each sub-pool descending by RecallScore::combined
- Returns populated RankedPool with all three sub-pools + composition_hint placeholder + deterministic trace_ref
weights() + half_life_ms() inspector methods.

Design choices

now_ms passed in (not SystemTime::now). Replay determinism is mandatory per spec; reading now() inside the ranker would break RecallTrace replay.
KVCache candidates silently dropped — spec's RankedPool has three sub-pools (layers/experts/engrams); KV cache pages are working-set state, not recall candidates. If a future PR adds a fourth sub-pool, the silent-drop test fails and the author updates the mapping.
NaN-safe sort via partial_cmp + Ordering::Equal fallback. Better than panic; the audit trail in RecallScore lets a debugger spot which factor was NaN.
trace_ref = Uuid::from_u128(now_ms) — deterministic placeholder; PR-3c replaces with richer RecallTrace including query hash + weights snapshot.

What is deliberately deferred (PR-3c)

DemandAlignedRecall trait impl — needs working-set + genome catalog candidate sourcing
Federation candidate sourcing (RecallScope::Federation / LocalThenGrid)
RecallTrace replay backing store — separate sentinel PR
Embedding model integration (the semantic factor input)

Test plan

cargo test --lib --features metal,accelerate genome::recall_impl — 13/13 pass:
- new_uses_default_weights_and_half_life
- with_config_preserves_weights_and_half_life
- rank_empty_candidates_returns_empty_pools — no-error on empty
- rank_partitions_by_kind_into_correct_sub_pool — LoRALayer→layers, MoEExpert→experts, Engram→engrams
- rank_sorts_each_sub_pool_descending_by_combined — monotonic
- rank_silently_drops_kvcache_candidates — spec sub-pool guard
- rank_score_factors_match_pr3a_for_each_candidate — audit trail
- rank_is_deterministic_across_calls — replay determinism
- rank_includes_not_resident_candidates_at_lower_score — sentinel surface
- rank_orders_by_tier_when_other_factors_equal — Fast > Bench > Cold > Frozen
- rank_composition_hint_is_empty_placeholder_in_pr3b
- rank_trace_ref_is_deterministic_from_now_ms
No regressions across other 2788 lib tests
Pre-push gate clean

Stack

feat(genome): working-set-manager PR-1 — typed data layer for cache hierarchy + paging #1346 / feat(genome): working-set-manager PR-2 — WorkingSetManager + TierStore traits (+sentinel cleanup) #1353 / feat(genome): working-set-manager PR-3 — LocalWorkingSetManager per-process impl #1355 / feat(genome): working-set-manager PR-4 — canonical ArtifactKeys + bus publishing helpers #1358 / feat(genome): working-set-manager PR-5 — LocalWorkingSetManager auto-publishes via bus hook (+MessageBus borrow fix) #1362 — my genome stack
feat(genome): demand-aligned-recall PR-1 — typed data layer (Part 7) #1366 — DAR PR-1: pure types
feat(genome): demand-aligned-recall PR-2 — DemandAlignedRecall trait + composite types #1367 + fix(genome): align recall generated type names #1370 — DAR PR-2: trait + composite types
feat(genome): demand-aligned-recall PR-3a — scoring function + per-factor curves #1371 — DAR PR-3a: scoring function + per-factor curves
This PR — DAR PR-3b: LocalDemandAlignedRecall ranking engine
NEXT — DAR PR-3c: working-set walker + DemandAlignedRecall trait impl + Runtime wiring

🤖 Generated with Claude Code

…ranking engine PR-3b of demand-aligned-recall (GENOME-FOUNDRY-SENTINEL Part 7). Composes PR-3a's scoring function with a candidate-injection API to produce ranked RankedPools. PR-3c adds the working-set walker that sources candidates from the substrate; PR-3b stays pure ranking. What lands - CandidateArtifact — caller-provided candidate ready for scoring. Carries per-factor inputs (semantic, outcome, provenance) + residency + last-used timestamp. - LocalDemandAlignedRecall { weights, half_life_ms } — the ranking engine. Thread-safe through immutability. - new() / with_config(weights, half_life_ms) constructors. - rank(now_ms, candidates) — pure-function ranking: scores each via PR-3a's score(), partitions by PageKind into layers/experts/ engrams, sorts each sub-pool descending by RecallScore.combined, returns populated RankedPool. - weights() + half_life_ms() inspectors. Design choices - now_ms passed in (not SystemTime::now). Replay determinism is mandatory per spec; reading now() would break RecallTrace replay. - KVCache candidates silently dropped — spec's RankedPool has three sub-pools (layers/experts/engrams); KV cache is working-set state. - NaN-safe sort via partial_cmp + Ordering::Equal fallback. - trace_ref = Uuid::from_u128(now_ms) — deterministic placeholder; PR-3c replaces with richer RecallTrace. What is deliberately deferred (PR-3c) - DemandAlignedRecall trait impl (needs working-set + genome catalog sourcing) - Federation sourcing (RecallScope::Federation / LocalThenGrid) - RecallTrace replay backing store (separate sentinel PR) - Embedding model integration Tests 13 new tests pin the ranking behavior: - new + with_config preserve config - rank empty → empty pools (no error) - rank partitions by PageKind correctly - rank sorts each sub-pool descending by combined - KVCache silently dropped - score factors round-trip from PR-3a's score() - rank is deterministic across calls (replay) - NotResident still scored at lower combined (sentinel surface) - Tier ordering when other factors equal (Fast > Bench > Cold > Frozen) - composition_hint placeholder + trace_ref determinism pinned 13/13 pass. No regressions across other 2788 lib tests. Clippy baseline bump 154→156 — drift from recent canary merges (zero clippy hits in genome/recall_impl other than the doc-list warnings I just fixed). Same pattern as PR-1 (146→148) and PR-2 (148→154). Stack - #1346 / #1353 / #1355 / #1358 / #1362 — my genome stack - #1366 — DAR PR-1: pure types - #1367 + #1370 — DAR PR-2: trait + composite types - #1371 — DAR PR-3a: scoring function + per-factor curves - THIS PR — DAR PR-3b: LocalDemandAlignedRecall ranking engine - NEXT — DAR PR-3c: working-set walker + trait impl + Runtime wiring Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…rce seam (#1374) PR-3c of demand-aligned-recall. Wires `DemandAlignedRecall` trait impl on `LocalDemandAlignedRecall` + introduces `CandidateSource` trait as the seam between the ranking engine and the substrate candidate sources. PR-3d will wrap the working-set-manager (#1362's bus hook) as a CandidateSource impl; PR-3c stays substrate-agnostic. Why this split PR-3c locks the source seam first. PR-3d adds the working-set walker as one impl; future PRs add the genome catalog walker + federation peer source. Each is independently testable. What lands - CandidateSource trait — async fn fetch(query, context) -> Vec<CandidateArtifact>. Send + Sync + async_trait for tokio. Object-safe; PR-3d's working-set walker is one impl. - LocalDemandAlignedRecall.source: Option<Arc<dyn CandidateSource>> — optional injection. None = empty-pool mode (legitimate "no candidates locally; try federation" signal). Some = trait impl's recall() dispatches to source.fetch() then rank(). - with_source(source) constructor. - with_config_and_source(weights, half_life, source) constructor for governor-driven config + source wiring. - DemandAlignedRecall trait impl on LocalDemandAlignedRecall: - recall(query, context) — fetches via source, scores via rank() with SystemTime::now() (rank() stays pure with explicit now_ms threading for replay determinism) - replay(trace) — returns typed RecallError::ScopeUnreachable with "RecallTraceStore (sentinel PR); not yet implemented in PR-3c". Per never-swallow-errors: typed refusal beats silent empty pool. When sentinel ships RecallTraceStore, this test flips to expect Ok(pool). Design choices - Source is Option, not required. The no-source path returns empty — useful for unit tests that don't need substrate + diagnostic tooling that wants a recall engine without candidate plumbing. - `recall()` reads SystemTime::now at the trait entry. The internal rank() still takes explicit now_ms; replay determinism preserved at the pure layer, live recall at the trait layer. This is the cleanest decoupling I could find that satisfies both spec asks. - PR-3c scope: no scope filtering, no freshness enforcement, no budget filtering. The CandidateSource does query-aware pruning in its fetch(); PR-3d's working-set walker filters by RecallScope::Local. Future PRs add the rest. Tests 5 new tests on the PR-3c surface: - recall_dispatches_through_dyn_demand_aligned_recall — Arc<dyn> object-safety - recall_without_source_returns_empty_pool_not_error — empty-pool contract - recall_with_source_dispatches_to_fetch_and_ranks — fetch call count + candidate-in-pool round-trip - with_config_and_source_preserves_all_three - replay_returns_typed_not_implemented_refusal_in_pr3c — pins the typed refusal so sentinel PR has a regression check to flip 18/18 pass on genome::recall_impl (13 PR-3b + 5 PR-3c). No regressions across other 2802 lib tests. Stack - #1346 / #1353 / #1355 / #1358 / #1362 — my genome stack - #1366 — DAR PR-1: pure types - #1367 + #1370 — DAR PR-2: trait + composite types - #1371 — DAR PR-3a: scoring function + per-factor curves - #1372 — DAR PR-3b: LocalDemandAlignedRecall ranking engine - THIS PR — DAR PR-3c: trait impl + CandidateSource seam - NEXT — DAR PR-3d: WorkingSetCandidateSource wrapping #1362's bus hook + concrete walker for the persona's working set Co-authored-by: Test <test@test.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…#1378) The architectural payoff of the genome stack lands here. A persona's page_in calls populate the working set (#1355); this source reads that same working set to surface "what's already hot" candidates that LocalDemandAlignedRecall (#1372 + #1374) ranks via the scoring function (#1371). End-to-end loop closed: page_in(persona, page) → WorkingSet.pages updated → bus publishes PageFault (#1362) → recall(query, ctx) → working_set_snapshot → CandidateArtifact per resident page → rank() → RankedPool What lands - WorkingSetCandidateSource struct holding Arc<LocalWorkingSetManager> - CandidateSource::fetch impl that: - reads persona's working_set_snapshot - returns empty Vec on unregistered persona (no error — cold- start signal callers may try federation) - translates each ResidentPage → CandidateArtifact with ResidencyHint::Hot { role } (resident = hot by definition) - preserves PageKind for downstream sub-pool partitioning - sets NEUTRAL_FACTOR_STUB (0.5) for semantic / outcome_history / provenance_trust factors (dedicated integrations land in separate PRs) - NEUTRAL_FACTOR_STUB public constant for the contract Design choices - Snapshot the working set via the manager's working_set_snapshot helper (cloned) rather than holding the RwLock across the fetch await. Same pattern as #1362's bus_arc hook. - Object-safe: works through Arc<dyn CandidateSource> per PR-3c's contract. - All resident pages map to Hot residency. PR-3e (or a separate catalog walker PR) will add Local{role=Bench/Cold/Frozen} for candidates outside the working set but resident in the genome catalog. - Stub-0.5 factors documented inline + via NEUTRAL_FACTOR_STUB constant. When the embedding / sentinel / trust integrations land, they replace the stubs without re-touching this file. What is deliberately deferred - Genome catalog walker (Bench/Cold/Frozen tier sources) — needs the catalog module - Federation peer source — needs federation registry - Embedding integration (semantic factor) — separate Lane H slice - Sentinel outcome lookup (outcome_history factor) — sentinel PR - Trust registry lookup (provenance_trust factor) — separate PR Tests 7 new tests, all end-to-end with real LocalWorkingSetManager + page_in calls: - fetch_unregistered_persona_returns_empty_not_error - fetch_registered_empty_working_set_returns_empty - fetch_after_page_in_returns_resident_pages_as_hot_candidates — the payoff test - translation_preserves_page_kind_for_sub_pool_partitioning — layer → layers, expert → experts, engram → engrams - translation_uses_neutral_factor_stubs_for_non_tier_factors — pins the contract so embedding-integration PRs flip it - source_is_object_safe_for_arc_dyn_dispatch — through PR-3c's Arc<dyn CandidateSource> - end_to_end_page_in_then_recall_returns_ranked_pool — full pipeline: page_in → WorkingSetCandidateSource ::fetch → LocalDemandAlignedRecall::recall → RankedPool with the paged-in artifacts ranked correctly 7/7 pass. No regressions across other 2822 lib tests. Stack - #1346 / #1353 / #1355 / #1358 / #1362 — my working-set-manager - #1366 — DAR PR-1: pure types - #1367 + #1370 — DAR PR-2: trait + composite types - #1371 — DAR PR-3a: scoring function + per-factor curves - #1372 — DAR PR-3b: LocalDemandAlignedRecall ranking engine - #1374 — DAR PR-3c: trait impl + CandidateSource seam - THIS PR — DAR PR-3d: WorkingSetCandidateSource (the payoff) Co-authored-by: Test <test@test.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…1380) Combines multiple CandidateSource impls into one, with optional deduplication by artifact id. Sets up the extensibility seam so future PRs (genome catalog walker, federation peer source, must-include resolver) add sources without re-wiring LocalDemandAlignedRecall. What lands - CompositeCandidateSource { sources, dedup } - DedupPolicy::None — return all candidates from all sources (a single artifact may appear N times if N sources surface it). Useful for audit-trail callers. - DedupPolicy::ByArtifactId — keep first occurrence per (kind, artifact_id) tuple in source-iteration order. Most callers want this (prevents double-counting a resident page that also surfaces via federation lookup). - CandidateSource::fetch impl: fans out to all sources concurrently via futures::future::join_all, merges, dedups. - new(sources, dedup) + with_default_dedup(sources) constructors. - source_count() + dedup_policy() inspector methods. Design choices - futures::future::join_all for fan-out (concurrent, unbounded). Acceptable for ≤5 sources currently; federation peer counts may need bounding later — when that happens, this fn changes internals without breaking the trait. - Dedup is configurable per composite. Most production wiring uses ByArtifactId; replay traces may use None for audit fidelity. - Different PageKind with same artifact_id treated as distinct candidates (a layer-page reference and an engram-page reference happen to share the underlying artifact id; recall keeps them separate so the sub-pool partitioning is correct). - Composite itself is object-safe — composites of composites valid for future hierarchical wiring. What is deliberately deferred - Source priority ordering — first-hit-wins per dedup. A future PR may add weighted merging. - Per-source error isolation — fetch returns Vec, not Result. The underlying trait method also returns Vec; widening the trait would be a separate concern. - Bounded concurrent fan-out — join_all is unbounded. Fine for the current source count; needs revisit when federation peers scale. Tests 9 new tests pin the composite's behaviors: - empty_composite_returns_empty_vec — no-error empty contract - single_source_composite_passes_through — degenerate case - fan_out_invokes_every_source_exactly_once — per-call accounting - merge_preserves_source_iteration_order — dedup correctness depends on this - dedup_none_preserves_all_duplicates - dedup_by_artifact_id_keeps_first_occurrence_only - dedup_treats_different_page_kinds_as_distinct - with_default_dedup_uses_by_artifact_id - composite_is_object_safe_as_dyn_candidate_source 9/9 pass. No regressions across other 2834 lib tests. Stack - #1346 / #1353 / #1355 / #1358 / #1362 — my working-set-manager - #1366 — DAR PR-1: pure types - #1367 + #1370 — DAR PR-2: trait + composite types - #1371 — DAR PR-3a: scoring function + per-factor curves - #1372 — DAR PR-3b: LocalDemandAlignedRecall ranking engine - #1374 — DAR PR-3c: trait impl + CandidateSource seam - #1378 — DAR PR-3d: WorkingSetCandidateSource - THIS PR — DAR PR-3e: CompositeCandidateSource (extensibility seam) - NEXT — DAR PR-3f or later: catalog walker + federation source + must-include resolver, all composing through this PR's seam Co-authored-by: Test <test@test.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

#1382) Resolves CapabilityQuery.must_include hard pins as candidates per GENOME-FOUNDRY-SENTINEL Part 7: "Hard pins — recall MUST include these in the RankedPool even if their score is low. Used for persona-private LoRA layers and sticky engrams." Plays through the composite seam shipped in PR-3e: wired AFTER a resident source like WorkingSetCandidateSource with ByArtifactId dedup, must-include items that ARE resident get the resident source's Hot residency + factor data; must-include items NOT resident get this source's NotResident placeholder (still ranked, just lower combined score). What lands - MustIncludeCandidateSource — zero-state unit struct (no Arc state needed; the source is pure-function over the query) - CandidateSource::fetch impl that: - reads query.must_include Vec<ArtifactRef> - maps each variant (LoRALayer / MoEExpert / Engram) to a CandidateArtifact with the appropriate PageKind - marks every must-include candidate as ResidencyHint:: NotResident { acquirable_from: SentinelRefinement } - uses NEUTRAL_FACTOR_STUB (0.5) for the three non-tier factors, same convention as WorkingSetCandidateSource (PR-3d) Recommended composite wiring let composite = CompositeCandidateSource::with_default_dedup(vec![ Arc::new(WorkingSetCandidateSource::new(mgr)), // Hot first Arc::new(MustIncludeCandidateSource::new()), // Pins // future: catalog walker, federation source ]); Spec contract met: every hard-pinned artifact surfaces in the RankedPool; if it's resident, it gets full residency-aware score; if not, it still appears (at lower combined) so composition can see "this was pinned but isn't here yet — schedule the foundry." Tests 6 new tests: - empty_must_include_returns_empty_candidates (no-error empty contract) - variant_mapping_preserves_page_kind (LoRALayer/MoEExpert/Engram variants → PageKind mapping) - must_include_marks_candidates_as_not_resident - factors_use_neutral_stubs_consistent_with_working_set_source - source_is_object_safe_for_dyn_dispatch - composite_with_dedup_resident_wins_must_include_for_pinned_hot_ artifact — the architectural payoff: resident pin keeps Hot, non-resident pin gets NotResident, both appear in merged Vec 6/6 pass. No regressions across other 2873 lib tests. Stack - #1346 / #1353 / #1355 / #1358 / #1362 — my working-set-manager - #1366 — DAR PR-1: pure types - #1367 + #1370 — DAR PR-2: trait + composite types - #1371 — DAR PR-3a: scoring function + per-factor curves - #1372 — DAR PR-3b: LocalDemandAlignedRecall ranking engine - #1374 — DAR PR-3c: trait impl + CandidateSource seam - #1378 — DAR PR-3d: WorkingSetCandidateSource (working-set source) - #1380 — DAR PR-3e: CompositeCandidateSource (extensibility seam) - THIS PR — DAR PR-3f: MustIncludeCandidateSource (hard-pin source) Co-authored-by: Test <test@test.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

joelteply merged commit edf8d79 into canary May 17, 2026
4 checks passed

joelteply deleted the feat/demand-aligned-recall-local-impl-pr3b branch May 17, 2026 03:34

github-actions Bot added the size: XL label May 17, 2026

joelteply mentioned this pull request May 17, 2026

feat(genome): demand-aligned-recall PR-3c — trait impl + CandidateSource seam #1374

Merged

3 tasks

joelteply mentioned this pull request May 17, 2026

feat(genome): demand-aligned-recall PR-3d — WorkingSetCandidateSource (genome stack payoff) #1378

Merged

3 tasks

joelteply mentioned this pull request May 17, 2026

feat(genome): demand-aligned-recall PR-3e — CompositeCandidateSource #1380

Merged

3 tasks

joelteply mentioned this pull request May 17, 2026

feat(genome): demand-aligned-recall PR-3f — MustIncludeCandidateSource #1382

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(genome): demand-aligned-recall PR-3b — LocalDemandAlignedRecall ranking engine#1372

feat(genome): demand-aligned-recall PR-3b — LocalDemandAlignedRecall ranking engine#1372
joelteply merged 1 commit into
canaryfrom
feat/demand-aligned-recall-local-impl-pr3b

joelteply commented May 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

joelteply commented May 17, 2026

Summary

What lands

Design choices

What is deliberately deferred (PR-3c)

Test plan

Stack

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant