Skip to content

Commit cf2362d

Browse files
jddunnjddunn
committed
spec: PerspectiveObserver — multi-agent subjective memory encoding
1 parent e1a3158 commit cf2362d

1 file changed

Lines changed: 388 additions & 0 deletions

File tree

‎docs/superpowers/specs/2026-04-10-perspective-observer-design.md‎

Lines changed: 388 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,388 @@
1+
---
2+
title: PerspectiveObserver — Multi-Agent Subjective Memory Encoding
3+
date: 2026-04-10
4+
status: draft
5+
scope: agentos
6+
depends_on:
7+
- 2026-04-10-memory-archive-rehydration-design
8+
enables:
9+
- 2026-04-XX-wilds-end-of-session-memory-pipeline-design
10+
---
11+
12+
# PerspectiveObserver — Multi-Agent Subjective Memory Encoding
13+
14+
## Problem Statement
15+
16+
AgentOS observation is single-agent. `MemoryObserver` extracts `ObservationNote[]` from conversation text, `ObservationCompressor` compresses them, and `ObservationReflector` derives patterns. All three operate on one agent's view of one conversation. There is no mechanism to encode the same event differently for multiple witnesses.
17+
18+
In multi-agent scenes (game NPCs, collaborative agents, simulated environments), every agent currently records the same objective text. HEXACO traits modulate *encoding strength* (how well the trace is remembered) and *retrieval behavior* (reconsolidation drift, FOK sensitivity), but the *content* of the memory is identical across agents. A cowardly NPC and a battle-hardened warrior remember the same fight with the same words. That's not subjectivity — it's identical records with different decay curves.
19+
20+
The fix: a `PerspectiveObserver` that takes an objective event and an array of witnesses, and produces per-witness first-person memory traces via LLM rewriting. Each witness's HEXACO traits, current mood, and relationships to entities in the event shape what they notice, how they feel about it, and what they remember.
21+
22+
Verified gap: zero matches for `witness|perspective|multi-agent|subjective|perAgent|forAgent` across `packages/agentos/src/memory` (searched 2026-04-10).
23+
24+
---
25+
26+
## Non-Goals
27+
28+
- **Not** replacing `MemoryObserver`, `ObservationCompressor`, or `ObservationReflector`. Those are single-agent conversation-analysis tools. `PerspectiveObserver` is a sibling with a different input shape and job.
29+
- **Not** modifying the core `MemoryTrace` interface. Perspective metadata goes in `structuredData.mechanismMetadata` like all other mechanism outputs.
30+
- **Not** supporting perspective encoding for `combatant` or `background` tier witnesses. Only `important`-tier witnesses get LLM rewrites; others fall back to objective encoding.
31+
- **Not** changing how events are detected or classified. The caller provides `ObservedEvent` objects — PerspectiveObserver does not extract events from text.
32+
- **Not** archiving objective events. That is `IMemoryArchive`'s job (Spec A, shipped). PerspectiveObserver consumes the archive reference to link subjective traces back to the archived original.
33+
34+
---
35+
36+
## Architecture
37+
38+
### Position in the Pipeline
39+
40+
```
41+
Turn Pipeline (wilds-ai)
42+
│
43+
├── Stage 7: Memory Persistence (existing)
44+
│ ├── persistMessage()
45+
│ ├── updateNarratorState()
46+
│ ├── updateGameState()
47+
│ │
48+
│ └── NPC Memory Observation (currently objective)
49+
│ │
50+
│ ├── [OLD] NpcMemoryBridge.observeForNpc(npcId, playerAction, narratorProse)
51+
│ │ → facade.observe('user', playerAction)
52+
│ │ → facade.observe('assistant', npcResponse)
53+
│ │ Result: OBJECTIVE text in NPC brain
54+
│ │
55+
│ └── [NEW] PerspectiveObserver.rewrite(event, witnesses)
56+
│ → archive.store(objectiveEvent) ← Spec A
57+
│ → LLM rewrite per witness (batched)
58+
│ → SubjectiveTrace per witness → witness brain
59+
│ Result: FIRST-PERSON text in each NPC brain
60+
```
61+
62+
`PerspectiveObserver` is a standalone pipeline stage in `packages/agentos/src/memory/pipeline/observation/`. It is a sibling to `ObservationCompressor` and `ObservationReflector`, not an extension of either. Different input shape (`ObservedEvent` + `Witness[]`), different output shape (`SubjectiveTrace[]`), different job (perspective rewriting vs note extraction vs pattern reflection).
63+
64+
### Types
65+
66+
```ts
67+
// packages/agentos/src/memory/pipeline/observation/PerspectiveObserver.ts
68+
69+
import type { PADState, HexacoTraits } from '../../core/config.js';
70+
import type { EmotionalContext } from '../../core/types.js';
71+
import type { IMemoryArchive } from '../../archive/IMemoryArchive.js';
72+
73+
/**
74+
* An objective event witnessed by one or more agents.
75+
* Provided by the caller (e.g. wilds-ai turn pipeline).
76+
*/
77+
export interface ObservedEvent {
78+
/** Unique event ID for linking subjective traces back to the source. */
79+
eventId: string;
80+
/** Objective event text (e.g. narrator prose + player action combined). */
81+
content: string;
82+
/** The player's action text. */
83+
playerAction: string;
84+
/** The narrator/system response text. */
85+
narratorProse: string;
86+
/** 0-1 importance score (from Director or NLP classification). */
87+
importance: number;
88+
/** PAD snapshot at the moment of the event. */
89+
emotionalContext: EmotionalContext;
90+
/** Entity names involved in the event. */
91+
entities: string[];
92+
/** When the event occurred (Unix ms). */
93+
timestamp: number;
94+
}
95+
96+
/**
97+
* A relationship between a witness and an entity in the event.
98+
* Used to bias the perspective rewrite (e.g. hostile toward the player
99+
* means interpreting their actions with suspicion).
100+
*/
101+
export interface WitnessRelationship {
102+
/** Entity name (must match an entity in the ObservedEvent). */
103+
entityName: string;
104+
/** Current disposition toward this entity. */
105+
disposition: 'neutral' | 'friendly' | 'wary' | 'hostile' | 'grateful' | 'fearful';
106+
/** Trust level, -1 (deep distrust) to 1 (full trust). */
107+
trustLevel: number;
108+
}
109+
110+
/**
111+
* An agent witnessing the event. Each witness gets their own
112+
* subjective trace via LLM perspective rewriting.
113+
*/
114+
export interface Witness {
115+
/** Agent/NPC ID — used to route the subjective trace to the correct brain. */
116+
agentId: string;
117+
/** Display name for the LLM prompt (e.g. "Lyra", "Guard Captain Holt"). */
118+
agentName: string;
119+
/** HEXACO personality traits for perspective bias. */
120+
hexaco: HexacoTraits;
121+
/** Current mood at observation time. */
122+
mood: PADState;
123+
/** Relationships to entities in the event. */
124+
relationships: WitnessRelationship[];
125+
/** Memory tier — only 'important' witnesses get LLM rewrites. */
126+
tier: 'important' | 'combatant' | 'background';
127+
}
128+
129+
/**
130+
* A first-person memory trace produced by perspective rewriting.
131+
* Routed to the witness's brain by the caller.
132+
*/
133+
export interface SubjectiveTrace {
134+
/** ID of the witness who produced this trace. */
135+
witnessId: string;
136+
/** First-person rewritten memory content. */
137+
content: string;
138+
/** Event ID linking back to the archived objective event. */
139+
sourceEventId: string;
140+
/** SHA-256 of the objective event content for integrity linking. */
141+
originalEventHash: string;
142+
/** Snapshot of the witness's state at encoding time. */
143+
perspectiveMetadata: {
144+
hexacoSnapshot: HexacoTraits;
145+
moodSnapshot: PADState;
146+
relationshipSnapshot: WitnessRelationship[];
147+
};
148+
}
149+
150+
/**
151+
* Result of a rewrite batch.
152+
*/
153+
export interface PerspectiveRewriteResult {
154+
/** Subjective traces successfully produced. */
155+
traces: SubjectiveTrace[];
156+
/** Witnesses that fell back to objective encoding due to LLM failure. */
157+
fallbacks: Array<{ witnessId: string; reason: string }>;
158+
/** Total LLM calls made. */
159+
llmCallCount: number;
160+
}
161+
```
162+
163+
### Gating Predicates
164+
165+
Events and witnesses are filtered before LLM calls. An event/witness pair is **skipped** (falls back to objective encoding) when any of these is true:
166+
167+
| Predicate | Rationale |
168+
|---|---|
169+
| `event.importance < 0.3` | Trivial events (walking, looking around) don't need subjective encoding — objective text is fine. |
170+
| `witness.tier !== 'important'` | Combatant and background NPCs get raw objective encoding. The LLM cost is reserved for narratively significant characters. |
171+
| `event.entities` has no overlap with `witness.relationships[].entityName` AND the witness is not named in the event | If the NPC has no relationship to anyone in the event and isn't mentioned, they're a passive bystander — objective encoding is sufficient. |
172+
173+
Skipped witnesses get the objective event text as-is, with `perspectiveEncoded: false` in their trace metadata. This is not a failure — it's a cost optimization.
174+
175+
### LLM Rewrite
176+
177+
**Model:** Haiku-class (cheap tier). The rewrite is a stylistic transform, not deep reasoning.
178+
179+
**Batching:** Up to 10 events per LLM call per witness. If a session turn produces 1 event, each witness gets 1 call. If an end-of-session consolidation produces 50 events, each witness gets 5 calls.
180+
181+
**Invoker contract:** `(system: string, user: string) => Promise<string>` — identical to `ObservationCompressor` and `ObservationReflector`.
182+
183+
**System prompt:**
184+
185+
```
186+
You are encoding memories for {agentName}. Rewrite each event as this character's
187+
first-person memory. What stands out to THEM? What do they notice, feel, emphasize?
188+
189+
Personality (HEXACO, 0-1 scale):
190+
- Honesty: {h} — low: spin things favorably; high: record things as they are
191+
- Emotionality: {e} — low: focus on facts; high: focus on feelings and atmosphere
192+
- Extraversion: {x} — low: internal monologue; high: focus on social dynamics
193+
- Agreeableness: {a} — low: note conflicts, competition; high: note cooperation
194+
- Conscientiousness: {c} — low: skip details; high: note commitments, consequences
195+
- Openness: {o} — low: stick to what happened; high: wonder about implications
196+
197+
Current mood: valence={v}, arousal={a}, dominance={d}
198+
199+
Relationships:
200+
{foreach relationship}
201+
- {entityName}: {disposition} (trust: {trustLevel})
202+
{end}
203+
204+
Rules:
205+
1. Write 1-2 sentences per event, first person.
206+
2. Personality MUST color the encoding — a suspicious character notices threats,
207+
an emotional character remembers how things felt, a conscientious character
208+
tracks who promised what.
209+
3. Hostile relationships mean interpreting actions with suspicion.
210+
4. Friendly relationships mean charitable interpretation.
211+
5. Do NOT fabricate events that didn't happen. Rewrite perspective, not facts.
212+
213+
Output a JSON array of strings, one per event. No explanation.
214+
```
215+
216+
**User prompt:**
217+
218+
```
219+
Events to encode:
220+
1. {event1.content}
221+
2. {event2.content}
222+
...
223+
```
224+
225+
**Response parsing:** JSON array of strings. Each string is one first-person trace. If parsing fails or array length doesn't match event count, fall back to objective encoding for all events in the batch with `perspective_failed: true`.
226+
227+
### Cost Envelope
228+
229+
| Parameter | Value |
230+
|---|---|
231+
| Model | Haiku 4.5 (~$0.25/MTok input, ~$1.25/MTok output) |
232+
| Tokens per rewrite | ~300 input, ~100 output |
233+
| Cost per rewrite | ~$0.0002 |
234+
| Rewrites per session (5 important NPCs, 50 turns, 0.5 gating rate) | 125 |
235+
| **Cost per session** | **$0.025** |
236+
| Batch size (events per LLM call) | 10 |
237+
| LLM calls per session | 25 (5 NPCs × 5 batches) |
238+
| At 1000 sessions/day | $25/day |
239+
240+
### Reconsolidation Interaction
241+
242+
When a trace has `perspectiveEncoded: true` in its `MechanismMetadata`, `applyReconsolidation()` halves the effective `driftRate`. The rationale: perspective encoding already shifted the memory from objective truth at encoding time. Applying full reconsolidation drift on every retrieval would compound the distortion.
243+
244+
The `maxDriftPerTrace` cap (default 0.4) still applies — this is a rate reduction, not an exemption. Perspective-encoded traces drift more slowly but to the same maximum.
245+
246+
Implementation: one check in `applyReconsolidation()`:
247+
248+
```ts
249+
const effectiveRate = meta.perspectiveEncoded ? config.driftRate * 0.5 : config.driftRate;
250+
```
251+
252+
### Failure Modes
253+
254+
| Failure | Behavior |
255+
|---|---|
256+
| LLM call fails for a witness batch | All events in that batch fall back to objective encoding for that witness. `perspective_failed: true` metadata flag. Logged at `warn`. |
257+
| LLM response is not valid JSON | Same as above — batch fallback. |
258+
| LLM returns wrong number of traces vs events | Same — batch fallback. |
259+
| Archive store fails for the objective event | PerspectiveObserver proceeds without archiving. The objective event is not lost (it's in the narrator transcript) but the archive link is broken. Logged at `warn`. |
260+
| All witnesses gated out for an event | No LLM calls made. Event still archived. No error. |
261+
262+
### MechanismMetadata Extensions
263+
264+
Add to `MechanismMetadata` in `mechanisms/types.ts`:
265+
266+
```ts
267+
/** PerspectiveObserver: trace was encoded through a persona lens. */
268+
perspectiveEncoded?: boolean;
269+
/** PerspectiveObserver: ID of the source objective event. */
270+
perspectiveSourceEventId?: string;
271+
/** PerspectiveObserver: SHA-256 of the source objective event content. */
272+
perspectiveSourceHash?: string;
273+
```
274+
275+
---
276+
277+
## Module Structure
278+
279+
| File | Responsibility |
280+
|---|---|
281+
| `src/memory/pipeline/observation/PerspectiveObserver.ts` | Core class: gating, batching, LLM rewriting, fallback |
282+
| `src/memory/pipeline/observation/perspective-prompt.ts` | System/user prompt builders (separated for testability) |
283+
| `src/memory/pipeline/observation/__tests__/PerspectiveObserver.test.ts` | Unit tests with mock LLM |
284+
| `src/memory/pipeline/observation/__tests__/perspective-prompt.test.ts` | Prompt builder tests |
285+
| `src/memory/mechanisms/types.ts` | Add `perspectiveEncoded`, `perspectiveSourceEventId`, `perspectiveSourceHash` |
286+
| `src/memory/mechanisms/retrieval/Reconsolidation.ts` | Add `perspectiveEncoded` rate halving |
287+
288+
---
289+
290+
## Documentation Plan
291+
292+
### TSDoc
293+
294+
Every new public symbol gets full TSDoc (imperative, `@param`, `@returns`, `@see`, `@example`). The `typedoc` → `agentos-live-docs` pipeline picks it up automatically.
295+
296+
### Docs files to update
297+
298+
| File | Update |
299+
|---|---|
300+
| `docs/memory/MEMORY_ARCHITECTURE.md` | Add PerspectiveObserver to the pipeline diagram and module table. |
301+
| `docs/memory/COGNITIVE_MECHANISMS.md` | Add "Perspective Encoding" row: first-person rewriting at encoding time, reconsolidation rate halving. |
302+
| `docs/memory/COGNITIVE_MEMORY_GUIDE.md` | Add "Multi-agent memory" section with worked example showing 2 NPCs witnessing the same event. |
303+
304+
### Blog
305+
306+
New post: "Perspective Observer: NPCs That Remember Differently" — short, focused on the cognitive science (subjective encoding, reconstructive memory) and the practical outcome (distinct NPC minds).
307+
308+
---
309+
310+
## Testing Plan
311+
312+
### Unit tests
313+
314+
`src/memory/pipeline/observation/__tests__/PerspectiveObserver.test.ts`:
315+
316+
- **Gating:** event below importance threshold → no LLM call, objective fallback
317+
- **Gating:** combatant witness → no LLM call, objective fallback
318+
- **Gating:** no entity overlap → no LLM call, objective fallback
319+
- **Rewrite:** 2 witnesses with different HEXACO → 2 different SubjectiveTraces
320+
- **Batching:** 15 events → 2 batches (10 + 5) per witness
321+
- **Fallback:** LLM returns invalid JSON → all events in batch fall back to objective
322+
- **Fallback:** LLM returns wrong count → batch fallback
323+
- **Archive link:** SubjectiveTrace.originalEventHash matches archived event hash
324+
- **Metadata:** perspectiveEncoded = true, sourceEventId set
325+
- **Cost tracking:** llmCallCount in result matches expected batch count
326+
327+
`src/memory/pipeline/observation/__tests__/perspective-prompt.test.ts`:
328+
329+
- System prompt includes HEXACO values
330+
- System prompt includes mood values
331+
- System prompt includes relationships with correct disposition/trust
332+
- User prompt formats events as numbered list
333+
334+
`src/memory/mechanisms/__tests__/retrieval.test.ts` (extend existing):
335+
336+
- Reconsolidation driftRate halved when `perspectiveEncoded: true`
337+
- Reconsolidation driftRate unchanged when `perspectiveEncoded` absent
338+
- maxDriftPerTrace cap still applies with halved rate
339+
340+
### Integration test
341+
342+
`tests/integration/memory/perspective-rewrite-roundtrip.test.ts`:
343+
344+
- Create 2 WildsMemoryFacade instances (2 NPCs) with different HEXACO
345+
- Create a PerspectiveObserver with a mock LLM that returns distinct rewrites
346+
- Feed one event with both NPCs as witnesses
347+
- Verify each NPC's brain contains different first-person content
348+
- Verify the objective event is in the archive (Spec A)
349+
350+
---
351+
352+
## Rollout
353+
354+
1. **Add `perspectiveEncoded` / `perspectiveSourceEventId` / `perspectiveSourceHash` to `MechanismMetadata`** — pure type addition, no runtime change.
355+
2. **Add reconsolidation rate halving** — 1-line change in `Reconsolidation.ts`, test in existing suite.
356+
3. **Create `perspective-prompt.ts`** — prompt builders, pure functions, fully testable in isolation.
357+
4. **Create `PerspectiveObserver.ts`** — core class with gating, batching, LLM calls, fallback.
358+
5. **Create unit + integration tests.**
359+
6. **Update docs.**
360+
7. **Wilds-ai adoption:** modify `NpcMemoryBridge.observeForNpc()` to route through `PerspectiveObserver` for important-tier NPCs.
361+
362+
Each step is independently mergeable.
363+
364+
---
365+
366+
## Resolved Design Decisions
367+
368+
### 1. Standalone vs extending MemoryObserver → **Standalone**
369+
370+
`MemoryObserver` extracts notes from *conversation text* (single-agent, text→notes). `PerspectiveObserver` rewrites *objective events* per-witness (multi-agent, event→first-person traces). Different input shape, different output shape, different job. Coupling them would add complexity without benefit.
371+
372+
### 2. LLM vs template for rewrites → **LLM (Approach A)**
373+
374+
Template-based rewrites (Approach B) produce formulaic output — HEXACO only modulates which keywords are emphasized, not genuine perspective shifts. The cost ($0.025/session on Haiku) is negligible. The whole point is genuine subjectivity; templates defeat that.
375+
376+
### 3. Archive the objective event → **Yes, via Spec A**
377+
378+
The caller archives the objective event via `IMemoryArchive.store()` before calling `PerspectiveObserver.rewrite()`. The archive uses `archiveReason: 'perspective_source'`. This requires adding `'perspective_source'` to the `ArchiveReason` union type in `IMemoryArchive.ts` (minor additive change to Spec A's shipped code). SubjectiveTraces carry `originalEventHash` for integrity linking. This is the caller's responsibility, not PerspectiveObserver's — the observer doesn't own the archive.
379+
380+
### 4. Reconsolidation clamping → **Halve driftRate, keep maxDriftPerTrace**
381+
382+
Perspective encoding shifts memory from objective truth at encoding. Reconsolidation shifts it again at retrieval. Two drift sources. Halving the retrieval drift rate means perspective-encoded traces evolve more slowly but aren't immune. The `maxDriftPerTrace` cap (0.4) still bounds total drift regardless.
383+
384+
## Remaining Open Questions
385+
386+
1. **Should PerspectiveObserver accept a `modelOverride` parameter?** Default is Haiku-class, but some consumers might want Sonnet for higher-quality rewrites on critical events. Proposed: yes, optional `model?: string` on the config. Resolve during implementation.
387+
388+
2. **Should the access-log (Spec A) record perspective-rewrite events?** When the archive stores the objective event for perspective encoding, should that count as a "rehydration" for retention purposes? Proposed: no — perspective source archival is a write path, not a read path. Only `rehydrate()` calls write access-log entries.

0 commit comments

Comments
 (0)