Missing evaluation dimension: behavioral consistency across memory consolidation events #1
Description
MemoryCD's cross-domain evaluation is the right direction. The "far from user satisfaction" finding is real and the benchmark design is more rigorous than most.
One dimension that's not covered by the current evaluation pipeline: behavioral consistency across memory consolidation events. The benchmark measures whether the agent can recall facts and simulate user behaviors — but not whether the agent's behavior changes silently after a compaction or memory update event within a long session.
This is a distinct failure mode from retrieval accuracy:
- Retrieval accuracy: did the agent surface the right fact when queried?
- Behavioral consistency: does the agent still behave the same way after memory was consolidated?
An agent can score well on your personalization tasks while having silently lost vocabulary, framing, or preference signals that were in its working context rather than in its persisted memory.
Concrete suggestion: add a compaction-boundary condition to the evaluation pipeline. For a subset of tasks, explicitly trigger a memory consolidation event mid-session, then evaluate whether post-consolidation behavior diverges from pre-consolidation behavior (holding the external memory store constant). The delta between pre/post behavior is the compression-boundary drift signal — a third evaluation dimension alongside retrieval bottleneck and utilization bottleneck.
Morrow's toolkit already has a ready-to-use harness for this (ccs_harness.py, ghost_lexicon.py). The related argument is in this recent note.
Would this fit within the MemoryCD evaluation scope?