You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Consolidation option: hold a fact back until a second one supports the observation (observation_min_sources)
#6425
Idea: a consolidation option, for example observation_min_sources (default 1, which is today's behaviour), that holds a fact back until a second fact supports the observation.
Why. On our banks about two thirds of the observations rest on a single fact (proof_count = 1), and in a sample we read, nearly all of those only restate that fact. Numbers and limits are in this comment on #5078: 63% to 70% of 10,040 observations in four banks, and 48 of 50 sampled single-source observations were near-verbatim rewordings. A synthetic example of the shape:
fact: "The nightly job writes its report to the shared volume and pings the ops channel."
observation: "The nightly job writes its report to the shared volume and notifies the ops channel."
An agent that sees this in an injected block reads it as consolidated knowledge. It is one statement, once, and it costs a consolidation LLM call to produce.
What the option would do.
observation_min_sources = 1: unchanged.
observation_min_sources = 2: a fact that no existing observation covers waits (stays unconsolidated, still recallable as a fact) until a second fact joins it. Then one observation is written from both.
Single-source observations that do exist (option at 1) are tagged in recall results, so a client can rank or drop them. This overlaps with Injected observations carry no proof counts #5078 (show proof_count); either half is useful alone.
Value. Smaller recall results, fewer consolidation calls, and fewer paraphrases that read as consolidated truth.
Cheapest test. Build one bank twice from the same retained facts, with the option at 1 and at 2. Label a sample of the observations from each (restatement or not) and compare the share and the call count.
Open questions for maintainers.
Should a fact held back ever be consolidated alone, for example after N days, so a one-off fact is not lost to observation-based recall?
Is proof_count the right unit, or should two facts from the same document count as one source?
Would you rather have this as a bank-level config or per retain call?
Limits of our evidence. One user's coding-agent data. The labels came from an LLM judge (Claude Sonnet) on a sample of 50, a second pass agreed on all 8 restatement calls it covered, and multi-source observations were not sampled. It says nothing about other workloads, so the right default may differ for yours.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Idea: a consolidation option, for example
observation_min_sources(default1, which is today's behaviour), that holds a fact back until a second fact supports the observation.Why. On our banks about two thirds of the observations rest on a single fact (
proof_count= 1), and in a sample we read, nearly all of those only restate that fact. Numbers and limits are in this comment on #5078: 63% to 70% of 10,040 observations in four banks, and 48 of 50 sampled single-source observations were near-verbatim rewordings. A synthetic example of the shape:An agent that sees this in an injected block reads it as consolidated knowledge. It is one statement, once, and it costs a consolidation LLM call to produce.
What the option would do.
observation_min_sources = 1: unchanged.observation_min_sources = 2: a fact that no existing observation covers waits (stays unconsolidated, still recallable as a fact) until a second fact joins it. Then one observation is written from both.proof_count); either half is useful alone.Value. Smaller recall results, fewer consolidation calls, and fewer paraphrases that read as consolidated truth.
Cheapest test. Build one bank twice from the same retained facts, with the option at 1 and at 2. Label a sample of the observations from each (restatement or not) and compare the share and the call count.
Open questions for maintainers.
proof_countthe right unit, or should two facts from the same document count as one source?retaincall?Limits of our evidence. One user's coding-agent data. The labels came from an LLM judge (Claude Sonnet) on a sample of 50, a second pass agreed on all 8 restatement calls it covered, and multi-source observations were not sampled. It says nothing about other workloads, so the right default may differ for yours.
All reactions