Observational Memory capability (observe/reflect with token thresholds)

[adtyavrdhn]

Summary

A memory system that passively observes conversation tokens and, when thresholds are reached, triggers observation (summarize recent messages) and reflection (compress observations) cycles — maintaining long-term context without explicit user intervention.

Motivation

For long-running coding sessions, conversation context grows until it exceeds the model's window. Observational Memory (OM) is a more sophisticated approach than simple truncation (#21 Compaction): it uses separate observer and reflector models to extract important information from expiring messages before they're removed. Mastra implements a full OM system with 12+ event types and configurable thresholds.

Code References

Mastra — OM configuration

// packages/core/src/harness/types.ts:246-255
interface HarnessOMConfig {
  defaultObserverModelId?: string;   // Model for observations
  defaultReflectorModelId?: string;  // Model for reflections
  defaultObservationThreshold?: number;  // Token threshold (default: 30K)
  defaultReflectionThreshold?: number;   // Token threshold (default: 40K)
}

Mastra — OM progress tracking

// packages/core/src/harness/types.ts:401-431
interface OMProgressState {
  status: 'idle' | 'observing' | 'reflecting';
  pendingTokens: number;
  threshold: number;
  thresholdPercent: number;
  observationTokens: number;
  reflectionThreshold: number;
  buffered: {
    observations: { status, chunks, messageTokens, projectedMessageRemoval, observationTokens };
    reflection: { status, inputObservationTokens, observationTokens };
  };
  generationCount: number;
  stepNumber: number;
  preReflectionTokens: number;
}

Mastra — OM events (12+ types)

// packages/core/src/harness/types.ts:641-725
| { type: 'om_status'; windows: { active, buffered }; recordId, threadId, stepNumber, generationCount }
| { type: 'om_observation_start'; cycleId, operationType, tokensToObserve }
| { type: 'om_observation_end'; cycleId, durationMs, tokensObserved, observationTokens, observations?, currentTask?, suggestedResponse? }
| { type: 'om_observation_failed'; cycleId, error, durationMs }
| { type: 'om_reflection_start'; cycleId, tokensToReflect }
| { type: 'om_reflection_end'; cycleId, durationMs, compressedTokens, observations? }
| { type: 'om_reflection_failed'; cycleId, error, durationMs }
| { type: 'om_buffering_start'; cycleId, operationType, tokensToBuffer }
| { type: 'om_buffering_end'; cycleId, operationType, tokensBuffered, bufferedTokens, observations? }
| { type: 'om_activation'; cycleId, operationType, chunksActivated, tokensActivated }
| { type: 'om_thread_title_updated'; cycleId, threadId, oldTitle?, newTitle }

Mastra — OM model switching

// packages/core/src/harness/harness.ts:966-1159
async switchObserverModel({ modelId }): Promise<void>;
async switchReflectorModel({ modelId }): Promise<void>;
getObservationThreshold(): number;
getReflectionThreshold(): number;

pydantic-ai design sketch

class ObservationalMemory(AbstractCapability[AgentDepsT]):
    """Passive observation + reflection memory system."""
    
    observer_model: Model | KnownModelName = 'gpt-4o-mini'
    reflector_model: Model | KnownModelName = 'gpt-4o-mini'
    observation_threshold: int = 30_000  # tokens
    reflection_threshold: int = 40_000   # tokens
    
    async def before_model_request(self, ctx, messages):
        token_count = count_tokens(messages)
        if token_count > self.observation_threshold:
            observations = await self._observe(messages)
            # Replace old messages with observation summary
        if self._observation_tokens > self.reflection_threshold:
            reflections = await self._reflect(self._observations)
            # Compress observations into reflections

Who has this

Framework	Feature	Details
Mastra	Full OM system	2-phase (observe+reflect), configurable models/thresholds, 12+ events
Claude Code	Auto memory	Saves important info to persistent memory files
OpenAI Agents SDK	Not present	Manual memory management
MemGPT/Letta	Tiered memory	Working/archival/recall memory with auto-management

Relationship

• Sub-issue of: [Tracking] Capabilities abstraction for pydantic-ai #1 (Tracking)
• Depends on: AbstractCapability + CombinedCapability + Agent integration #10 (AbstractCapability), Expose context window size on ModelProfile and usage on RunContext #35 (Context window on ModelProfile)
• Related: Compaction capability #21 (Compaction — simpler version), Memory capability #30 (Memory capability)
• Blocked by: Expose context window size on ModelProfile and usage on RunContext #35 (needs context window size for threshold calculations)

[adtyavrdhn]

API dependencies

Observational Memory has two API-level dependencies:

1. Run-level hooks (#20) — OM needs to initialize state (load previous observations, set thresholds) at run start and persist state at run end. The current per-model-call hooks are insufficient — OM shouldn't reload from storage on every LLM call.

2. Context window size (#35) — OM's threshold calculations require knowing the model's context window to set meaningful observation/reflection thresholds. Mastra defaults to 30K/40K tokens, but this should be relative to the model's capacity. RunContext already exposes model, but model.profile().context_window needs to be available.

3. Capability state persistence — OM accumulates observations and reflections across steps within a run. The current stateless capability API requires this state to live in deps, meaning the user must wire it up externally. An on_run_start hook (#20) would provide a natural initialization point.

These are API design issues, not OM-specific issues — they affect #30 (Memory), #46 (Cost Budget), and #65 (Task Tracking) similarly.

[ferhimedamine]

The token-threshold OM approach (observe at 30K, reflect at 40K) is clean for cost-predictable pipelines, but creates cliff effects in practice — at 29K tokens everything is retained, at 30K the observer compresses aggressively, and the quality of what gets kept depends on a single-pass decision under time pressure.

An alternative pattern we've been running in production: weight each stored memory by importance and let continuous decay determine what persists. Each memory's effective importance = base_importance × access_recency_factor, so frequently-referenced context stays hot while one-off observations naturally fade. Consolidation is continuous rather than threshold-triggered — no cliff effects, no threshold tuning.

The tradeoff is explicit: OM gives precise cost control (you know exactly when compression runs and can budget for it), while decay-based approaches give better recall quality for long-running agents where what's "important" shifts unpredictably over the session. Past ~500 interactions, threshold-based systems struggle to pick the right observation window as session character changes. With decay, the system adapts automatically.

We tested this against 1,540 temporal reasoning questions (the LoCoMo benchmark) and found that decay-weighted retrieval significantly outperforms fixed-window approaches on multi-hop and event-ordering queries where the relevant context spans across observation boundaries. Implementation details and benchmark methodology: https://github.com/Dakera-AI/dakera-deploy