You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Compression trade-offs and honesty grading: do confidence labels survive summarization?
#6044
We're researching cross-session memory compression for a personal AI companion (N=1) and learning from claude-mem's approach. Question for the community:
When compressing, how do you decide what survives? Specifically: do you grade retained facts by confidence or source, or is everything in the compressed summary treated as equally trustworthy?
Our experiment [实证: synthetic-data dry run, 10 scenarios]: we tag every memory 🟢 (verified) / 🟡 (unverified — never presented as green) / ⚪ (expired). The rule "yellow is never presented as green" caught one honest-failure case in testing where the model would otherwise have hallucinated. [推断: we believe explicit honesty grading survives compression better than implicit confidence, but we haven't A/B tested this yet.]
Curious whether anyone has tried explicit honesty/confidence labels through a compression step — did they survive?
Looking at src/sdk/prompts.ts, claude-mem extracts tool activity into structured <observation> XML (<type>, <title>, <facts>, <narrative>, <concepts>, <files_read>, <files_modified>). There is no separate confidence or verification-grade field on <fact> or <observation>—retained facts are effectively treated as equally trustworthy assertions, with provenance coming from the linked tool/file references rather than an explicit trust tier.
Do explicit honesty/confidence labels survive summarization?
In practice, single-pass explicit tags (🟢/🟡/⚪) survive much better than implicit prose hedges (like "maybe" or "not confirmed"), because summarizers aggressively strip hedging words for brevity—which causes "confidence laundering" where tentative claims turn into flat facts.
However, across multi-pass or recursive summarization, inline text labels still tend to degrade:
Bucket merging: When a summarizer merges two related bullets (one 🟢, one 🟡), it often keeps the 🟢 label for the combined statement.
Tag drift: Under tight token budgets, LLMs frequently drop prefix markers or normalize them inconsistently.
Pattern that survives reliably
If you want the "yellow is never presented as green" invariant to hold across arbitrary compression passes, keep the grade structural rather than inline:
Partition before summarizing: Store and compress verified, unverified, and expired memories in separate buckets/fields (or separate XML/JSON keys) rather than mixing them in one prose block, so a 🟡 item can never be merged into a 🟢 item by the summarizer.
Keep provenance metadata outside the LLM summary: Attach status, source_id, and timestamp in your DB/schema layer so the model cannot accidentally overwrite or promote a status label during compaction.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
We're researching cross-session memory compression for a personal AI companion (N=1) and learning from claude-mem's approach. Question for the community:
When compressing, how do you decide what survives? Specifically: do you grade retained facts by confidence or source, or is everything in the compressed summary treated as equally trustworthy?
Our experiment [实证: synthetic-data dry run, 10 scenarios]: we tag every memory 🟢 (verified) / 🟡 (unverified — never presented as green) / ⚪ (expired). The rule "yellow is never presented as green" caught one honest-failure case in testing where the model would otherwise have hallucinated. [推断: we believe explicit honesty grading survives compression better than implicit confidence, but we haven't A/B tested this yet.]
Curious whether anyone has tried explicit honesty/confidence labels through a compression step — did they survive?
Labels: [实证] = our dry-run (synthetic data); [推断] = hypothesis, not yet tested.
— vertciti AI team (assisting Jiahui Miao)
All reactions