memory-plimpsest

Layered memory with ghost traces — write, fade, and read through layers like an ancient palimpsest

What is a Palimpsest?

A palimpsest is an ancient manuscript where the original writing was scraped off and new text was written over it — but traces of the older text remain visible beneath the surface. Archaeologists and historians can read through the layers to recover lost knowledge.

This crate brings that metaphor to agent memory. Instead of a simple key-value store where writes overwrite previous values, memory-plimpsest maintains a stack of semi-transparent layers. Each new write adds a layer on top. Older layers gradually fade (decay in opacity) but remain readable — as "ghost traces" — until they eventually decay completely.

This model is particularly suited for AI agents and cognitive systems where:

Recent context is most important (top layers, full opacity)
Past context shouldn't vanish instantly (ghost layers, faded but visible)
Very old context naturally fades away (fully decayed layers, pruned)

Why Does This Matter?

Traditional memory systems face a tension between persistence and relevance. Keep everything and you drown in stale data. Overwrite aggressively and you lose important context. The palimpsest model resolves this:

Gradual decay: Information doesn't disappear in a binary on/off — it fades, giving downstream systems a chance to rescue it
Read-through: Access the most recent state instantly, or drill down to historical layers
Ghost traces: Even faded layers leave imprints — useful for pattern detection and context recovery
Automatic pruning: Fully-decayed layers are removed without manual cleanup

Real-world applications:

Agent memory: Maintain conversation context that fades naturally as topics shift
Sensor data: Recent readings are sharp, older readings fade but remain comparable
Edit history: Track document revisions with natural emphasis on recent changes
Cognitive modeling: Simulate human-like memory decay curves in AI systems

Architecture

┌──────────────────────────────────────────────────────────────┐
│                  Memory Palimpsest Model                      │
│                                                              │
│  Write Stack (newest on top, oldest on bottom)               │
│                                                              │
│  Layer 3 [t=300] ████████████ opacity: 1.00  ← newest       │
│  Layer 2 [t=200] ████████░░░░ opacity: 0.72                 │
│  Layer 1 [t=100] ████░░░░░░░░ opacity: 0.36    ← fading     │
│  Layer 0 [t=50]  ██░░░░░░░░░░ opacity: 0.12    ← ghost      │
│                                                              │
│          │                                                   │
│          ▼ Decay: opacity *= decay_factor each write         │
│          ▼ Prune: remove layers with opacity → 0             │
│                                                              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐       │
│  │  ReadThrough │  │  GhostTrace  │  │  LayerDecay  │       │
│  │  read_top()  │  │  Faded but   │  │  Configurable│       │
│  │  drill_down()│  │  still there │  │  decay rate  │       │
│  │  search()    │  │  ~content~   │  │  + min opacity│      │
│  │  composite() │  │              │  │              │       │
│  └──────────────┘  └──────────────┘  └──────────────┘       │
└──────────────────────────────────────────────────────────────┘

Quick Start

use memory_plimpsest::{MemoryPlimpsest, LayerDecay, ReadThrough};

// Create a palimpsest with custom decay (opacity ×0.7 per new write)
let decay = LayerDecay::new(0.7, 0.01);
let mut palimpsest = MemoryPlimpsest::new(decay);

// Write layers — each new write decays all existing layers
palimpsest.write("Agent started monitoring emails", 1000);
palimpsest.write("Found 3 urgent messages", 2000);
palimpsest.write("Responded to urgent thread", 3000);

// Read the top (most recent) layer
let rt = ReadThrough::new(&palimpsest);
assert_eq!(rt.read_top(), Some("Responded to urgent thread"));

// Drill down through visible layers (opacity > 0.5)
let visible = rt.drill_down(0.5);
println!("Visible layers: {:?}", visible);

Searching and Composites

// Search for specific content (searches from newest to oldest)
let result = rt.search("urgent");
if let Some((layer_idx, content)) = result {
    println!("Found '{}' at layer {}", content, layer_idx);
}

// Get a composite view of all layers
let composite = rt.composite();
println!("Full context: {}", composite);
// Output: "Agent started... / [ghost: Found 3...] / Responded to..."

Working with Ghost Traces

// After several writes, older layers become ghosts (opacity < 0.5)
palimpsest.write("New task assigned", 4000);
palimpsest.write("Task completed", 5000);

let ghosts = palimpsest.ghost_traces();
for ghost in &ghosts {
    println!("Ghost from layer {}: {}", ghost.layer_index, ghost.render());
    // "~Agent started monitoring emails~" or "[too faded to read]"
}

Pruning Decayed Layers

// Remove fully-decayed layers to reclaim memory
let pruned_count = palimpsest.prune();
println!("Pruned {} fully-decayed layers", pruned_count);
println!("Active layers: {}", palimpsest.layer_count());

API Reference

PlimpsestLayer

Field	Type	Description
`content`	`String`	Text content of this layer
`timestamp`	`u64`	Unix timestamp when written
`opacity`	`f64`	Current opacity (0.0–1.0, decays over time)

Method	Returns	Description
`PlimpsestLayer::new(content, timestamp)`	`PlimpsestLayer`	Create layer at full opacity
`PlimpsestLayer::with_opacity(content, timestamp, opacity)`	`PlimpsestLayer`	Create layer at given opacity
`layer.is_ghost()`	`bool`	True if opacity < 0.1