|
| 1 | +# Tuning memory retrieval |
| 2 | + |
| 3 | +Fabri retrieves guidelines from memory and injects the most relevant ones into |
| 4 | +each run's prompt. The defaults are eval-backed and good for most stores — **if |
| 5 | +you're just starting out, change nothing.** This guide is for when you want to |
| 6 | +tune retrieval to your own corpus, and it shows the exact knobs, sane values, |
| 7 | +and how to *measure* whether a change helped instead of guessing. |
| 8 | + |
| 9 | +Every knob lives under the `memory:` block of your agent config (or the global |
| 10 | +`DEFAULT_CONFIG`). All are optional; omitting one keeps its default. |
| 11 | + |
| 12 | +> **Golden rule: never flip a retrieval default blind.** Fabri ships an offline |
| 13 | +> eval (`python -m fabri.benchmarks.retrieval_eval`) precisely so retrieval |
| 14 | +> changes move a *number*, not a vibe. Measure before and after. |
| 15 | +
|
| 16 | +--- |
| 17 | + |
| 18 | +## Start here: the default is `hybrid`, and why |
| 19 | + |
| 20 | +```yaml |
| 21 | +memory: |
| 22 | + retrieval_strategy: hybrid # the default — you can delete this line |
| 23 | +``` |
| 24 | +
|
| 25 | +On fabri's labeled eval fixture (40 guidelines / 24 queries, top_k=5), `hybrid` |
| 26 | +is the best strategy on **every** metric: |
| 27 | + |
| 28 | +| strategy | recall@1 | recall@3 | recall@5 | MRR | |
| 29 | +|---|---|---|---|---| |
| 30 | +| dense | 0.583 | 0.688 | 0.792 | 0.790 | |
| 31 | +| sparse | 0.500 | 0.792 | 0.875 | 0.772 | |
| 32 | +| **hybrid (default)** | **0.583** | **0.896** | **0.938** | **0.844** | |
| 33 | +| hybrid+mmr | 0.583 | 0.625 | 0.729 | 0.804 | |
| 34 | + |
| 35 | +`hybrid` fuses dense (vector) and sparse (BM25) retrieval, and **degrades |
| 36 | +gracefully to dense** wherever BM25 is unavailable (a Qdrant store without the |
| 37 | +`fabri[bm25]` extra), so it is never worse than plain `dense`. That's why it's a |
| 38 | +safe unconditional default. |
| 39 | + |
| 40 | +--- |
| 41 | + |
| 42 | +## The four strategies — when to pick each |
| 43 | + |
| 44 | +```yaml |
| 45 | +memory: |
| 46 | + retrieval_strategy: dense # | sparse | hybrid | hybrid+mmr |
| 47 | +``` |
| 48 | + |
| 49 | +| strategy | what it does | reach for it when | |
| 50 | +|---|---|---| |
| 51 | +| `hybrid` | RRF fusion of vector + BM25 | **default.** Best all-round; keyword + semantic. | |
| 52 | +| `dense` | vector similarity only | your queries are paraphrases with little keyword overlap, or you want the lowest-dependency path. | |
| 53 | +| `sparse` | BM25 keyword only | your guidelines are keyword/identifier-heavy (tool names, error codes, API paths) and you don't want vector cost. | |
| 54 | +| `hybrid+mmr` | hybrid, then diversify the final set | your store has many **near-duplicate** guidelines and the injected set keeps repeating the same point. MMR trades a little recall for diversity — measure first. | |
| 55 | + |
| 56 | +--- |
| 57 | + |
| 58 | +## Knob reference (with recommended values) |
| 59 | + |
| 60 | +### `rrf_k` — RRF fusion sharpness (hybrid only) |
| 61 | + |
| 62 | +```yaml |
| 63 | +memory: |
| 64 | + rrf_k: 20 # default |
| 65 | +``` |
| 66 | + |
| 67 | +RRF scores each candidate `Σ 1/(rrf_k + rank)`. The classic web-scale default is |
| 68 | +`60`, but fabri fuses **two short pools** (~2·top_k each), where `60` flattens |
| 69 | +the rank term so much that "appears in both lists" outranks "is the single best |
| 70 | +match." Retuning `60 → 20` lifted hybrid **recall@3 0.60 → 0.90** with recall@5 |
| 71 | +unchanged. |
| 72 | + |
| 73 | +- **Lower (10–20):** sharper — the top hit of either signal keeps its rank. |
| 74 | + Best for small/medium stores. |
| 75 | +- **Higher (40–60):** rewards agreement between signals over peak relevance. |
| 76 | + Only worth it for very large stores where both pools are long. |
| 77 | + |
| 78 | +### `temporal_decay` / `temporal_half_life_days` — prefer recent guidelines |
| 79 | + |
| 80 | +```yaml |
| 81 | +memory: |
| 82 | + temporal_decay: true # default false |
| 83 | + temporal_half_life_days: 30.0 # a 30-day-old entry is weighted ~0.5 |
| 84 | +``` |
| 85 | + |
| 86 | +Turn on when your domain **drifts** (APIs change, last month's lesson is stale). |
| 87 | +Leave off when guidelines are timeless. Shorter half-life = more aggressive |
| 88 | +recency bias. |
| 89 | + |
| 90 | +### `importance_weight` — boost proven guidelines |
| 91 | + |
| 92 | +```yaml |
| 93 | +memory: |
| 94 | + importance_weight: 0.2 # default; 0 disables |
| 95 | +``` |
| 96 | + |
| 97 | +Boosts entries by how often they've been retrieved plus a bonus for `strategic` |
| 98 | +ones: `score *= 1 + importance_weight * importance`. Raise (→0.4) to lean harder |
| 99 | +on battle-tested guidelines; set `0` for pure relevance. |
| 100 | + |
| 101 | +### `domain_routing` — soft-boost same-domain guidelines |
| 102 | + |
| 103 | +```yaml |
| 104 | +memory: |
| 105 | + domain_routing: true # default false |
| 106 | +``` |
| 107 | + |
| 108 | +A zero-cost keyword classifier tags each query as code/planning/search/api/ |
| 109 | +generic and gives same-domain entries a 1.15× boost (never a hard filter). Turn |
| 110 | +on when one agent spans clearly different domains and cross-domain guidelines |
| 111 | +leak in. |
| 112 | + |
| 113 | +### `mmr_lambda` — relevance vs diversity (hybrid+mmr only) |
| 114 | + |
| 115 | +```yaml |
| 116 | +memory: |
| 117 | + retrieval_strategy: hybrid+mmr |
| 118 | + mmr_lambda: 0.7 # 1.0 = pure relevance, 0.0 = pure diversity |
| 119 | +``` |
| 120 | + |
| 121 | +Only applies to `hybrid+mmr`. Lower it toward 0.5 if the injected set is |
| 122 | +redundant; keep ≥0.7 so relevance still dominates. |
| 123 | + |
| 124 | +### `top_k` — how many guidelines get injected |
| 125 | + |
| 126 | +```yaml |
| 127 | +memory: |
| 128 | + top_k: 5 # default |
| 129 | +``` |
| 130 | + |
| 131 | +More guidelines = more context (and tokens) per turn. Raising `top_k` trades |
| 132 | +prompt cost for coverage; the recall@k numbers above are at `top_k=5`. |
| 133 | + |
| 134 | +--- |
| 135 | + |
| 136 | +## About the `success_pattern` guarantee (no knob — just so you know) |
| 137 | + |
| 138 | +Fabri always reserves up to `top_k // 2` slots for `success_pattern` |
| 139 | +guidelines ("what worked last time"), so learned strategies surface even when a |
| 140 | +query doesn't obviously match one. These slots are **back-loaded** — the most |
| 141 | +relevant guideline always keeps rank 1, and success patterns fill reserved |
| 142 | +*tail* slots. (Front-loading them, an earlier bug, sank recall@1 from 0.58 to |
| 143 | +0.13.) You don't configure this; it's just why you may see a strategy guideline |
| 144 | +in the injected set that wasn't the closest match. |
| 145 | + |
| 146 | +--- |
| 147 | + |
| 148 | +## Recipes |
| 149 | + |
| 150 | +**"My store is tiny (a fresh agent)."** Defaults are fine. Retrieval short- |
| 151 | +circuits entirely on an empty store (no model load), so cold starts are cheap. |
| 152 | + |
| 153 | +**"I want the cheapest / most self-contained path."** `dense` with the `sqlite` |
| 154 | +backend — no Qdrant, no `fabri[bm25]`, no BM25 index. |
| 155 | + |
| 156 | +**"My guidelines are full of tool names / error codes / paths."** `sparse` or |
| 157 | +keep `hybrid` and lower `rrf_k` to ~10 so exact keyword hits rank first. |
| 158 | + |
| 159 | +**"The same advice keeps repeating in the prompt."** `hybrid+mmr` with |
| 160 | +`mmr_lambda: 0.6`. Measure — MMR can cost recall. |
| 161 | + |
| 162 | +**"My domain changes fast."** `temporal_decay: true`, |
| 163 | +`temporal_half_life_days: 14`. |
| 164 | + |
| 165 | +**"Large store (100k+ entries), fusion feels muddy."** Try `rrf_k: 40` and |
| 166 | +consider `domain_routing: true`. |
| 167 | + |
| 168 | +--- |
| 169 | + |
| 170 | +## Measure your tuning |
| 171 | + |
| 172 | +Run the head-to-head eval — it drives the **real** retrieval path, so the |
| 173 | +numbers reflect production: |
| 174 | + |
| 175 | +```bash |
| 176 | +python -m fabri.benchmarks.retrieval_eval # markdown table |
| 177 | +python -m fabri.benchmarks.retrieval_eval --strategies dense,hybrid --json |
| 178 | +python -m fabri.benchmarks.retrieval_eval --fixture my_corpus.json --top-k 8 |
| 179 | +``` |
| 180 | + |
| 181 | +To tune on **your** data, copy `tests/fixtures/retrieval_eval.json` and replace |
| 182 | +`corpus` (your guidelines) + `queries` (tasks tagged with the guideline |
| 183 | +label(s) that *should* surface). Keep guideline text unique within a `kind` |
| 184 | +(identical text collapses under the store's id hash). Then compare strategies / |
| 185 | +`rrf_k` values on your own numbers. |
| 186 | + |
| 187 | +The CI gate in `tests/test_retrieval_eval_gate.py` locks the shipped defaults to |
| 188 | +`measured − 0.05` so a future change can't silently regress them. |
| 189 | + |
| 190 | +## Debug what retrieval actually decided |
| 191 | + |
| 192 | +Every retrieval emits one structured `retrieval` trace event (strategy, dense/ |
| 193 | +sparse pool sizes, whether BM25 fired or silently fell back to dense, per- |
| 194 | +candidate `inclusion_reason`, MMR). It's trace-only (zero prompt cost). Read it |
| 195 | +straight from the run's JSONL trace to see *why* a guideline was or wasn't |
| 196 | +injected — the fastest way to tell "hybrid is secretly running as dense" (look |
| 197 | +for `sparse_fallback: true`) from a genuine relevance miss. |
| 198 | + |
| 199 | +See `docs/design/memory-observability-plan.md` for the event contract and |
| 200 | +`BENCHMARKS.md` for the full eval findings. |
0 commit comments