RFC: Quality model — separating human rating from machine-computed score (from the rating/quality arc)

[filhocf]

Routing this here as a design RFC, following the same pattern as the recent ones. Full draft lives at docs/rfc/rfc-quality-model.md on my fork; this is the summary for discussion.

Problem

quality_score is a single overloaded field. Six sources write to it (AI scorer, implicit scorer, async batch scorer, manual rating via MCP + HTTP twins, decay association-boost, Milvus conflict resolution) and ten consumers read it (decay multiplier, forgetting retention, four rerank weights, graph sort, analytics tiers, bootstrap ranking) — none can tell where the number came from.

The concrete failure: when a human rates a memory, the rating is blended (0.6*user + 0.4*existing) into the same field the machine computes. Human opinion and machine guess overwrite each other — the rating gets diluted or lost, or the machine's later score erases the human's judgment. One field, two authors scribbling over each other.

Empirical state on my deployment (22,709 memories): 246 have quality_score (1.1%), zero have quality_provider, 138 have user_rating. In practice effective quality today is human rating + a 0.5 default — the AI scorer has effectively never persisted here. So the overload is currently masked, but it becomes active the moment the scorer runs and then a rating lands on top.

Proposal (decisions taken, open questions below)

• Two origin fields, not one. computed_quality (machine, keeps quality_provider/quality_components) and user_rating (human) as distinct metadata fields, each preserving its own history. Additive — no schema change.
• Human opinion is authoritative. If a user rating exists, it wins for ranking and forgetting; the computed score is the fallback when there's no rating. Override-with-preservation, not a blend — the 0.6/0.4 blend is removed.
• A 👎 de-ranks, it doesn't destroy. A single negative rating lowers the memory in search but does not trigger aggressive forgetting — because "the human disapproves of this content" ≠ "this is unimportant" (a hard NEVER do X lesson may earn a 👎 yet must survive). Accumulated 👎 do lead to forgetting.
• Effective quality is materialized on write, not projected on read. ~7 consumers read metadata.get('quality_score') raw (including a ranking path and two that operate on plain dicts with no Memory object), so a property-based projection would miss them. Writing the composed value into quality_score at rate/score time means every existing reader gets it with zero consumer changes. The origin fields stay separate for reversibility.
• Provenance honesty. quality_provider stays strictly "which engine computed"; rating provenance lives in its own field, never masquerading as a provider (which would silently degrade to implicit in the sync codec's closed vocabulary).

Open questions I'd like your read on

1. Rating → quality normalization. Rating is -1/0/+1. How should it map onto the 0..1 scale decay/forgetting/ranking consume? (rating+1)/2 makes 👎 = 0.0 exactly, which under "human wins" drops it to the retention floor. A floor above 0 for 👎? A dedicated curve? This feeds the retention thresholds you own.
2. Accumulation threshold for negative-rating forgetting. Single 👎 de-ranks, multiple forget — but what's the exact criterion from rating_history (N distinct negatives? net-negative sum? window?) to fit forgetting.py?
3. Materialize vs read-time projection — acceptable to write the effective value into quality_score, given the raw-metadata readers?
4. Sync codec — compress_metadata_for_sync currently drops user_rating; to sync the human opinion across hosts it needs a positional CSV field gated by a codec version bump. OK to coordinate that?

Prior art I'm building on, not reinventing: the self-service RFC's "confidence + provenance first-class, separate from the raw score" (§4), and the original_quality_before_boost preservation precedent in consolidation/decay.py. Happy to split implementation into small PRs once the shape is agreed. This depends on landing before the AI scorer is switched on, so computed and rating don't fight.

[doobidoo]

Answered this together with #1312, since the two are the same arc and the questions there are the concrete end of it: #1312

Short version of the position, so this thread carries it too: keep the two origin fields, materialize the effective value into quality_score, and -- the part that is not in either write-up -- have retention read computed_quality rather than the effective value. Search ranking is where a human rating belongs, because a wrong down-vote there costs one bad ranking. forgetting.py is not reversible in the same way, and a thumbs-down is a statement about usefulness to one person at one moment.

Design discussion continues on #1312.