MindSpan: A 20-Year Longitudinal Cognitive Memory Benchmark and Why It Supersedes LongMemEval

[sbharatjoshi]

MindSpan: A 20-Year Longitudinal Cognitive Memory Benchmark — and Why It Supersedes LongMemEval

TL;DR

We're introducing MindSpan, a benchmark that spans 20+ years of coherent autobiographical memory for a single persona, with 19,512 memory records, a 4-generation kinship tree, full neuromorphic metadata (valence, arousal, importance), and 200 queries across 17 cognitive tracks. Spector Memory achieves 100% QA accuracy on MindSpan under a strict <1,800 token retrieval budget.

In contrast, LongMemEval (ICLR 2025) — the current academic standard — has an average target fact age of just 16.4 days, compresses 16 sessions per calendar day, and uses scraped multi-user transcripts as distractors. It tests short-term buffer retrieval, not long-term cognitive memory.

---

1. What is MindSpan?

MindSpan is a 20-year longitudinal cognitive benchmark designed to evaluate AI memory systems against the full complexity of human autobiographical memory. It's built from first cognitive science principles — not from scraped chat logs.

The Persona: Mike Thompson

The entire corpus follows a single, deeply-characterized persona:

• Mike Thompson, 36→38 years old (2024–2026), Principal Product Manager at Vertex Health in Frisco, TX
• Wife Sarah (Lead UX Designer), son Ethan (8→10), daughter Lily (3→5), Golden Retriever Cooper
• AI companion Jarvis (Spector Memory)

Two Temporal Epochs

Epoch	Time Span	Content
Autobiographical Foundations	1990–2023 (33 years)	~468 historical milestones: childhood memories, UIUC dorm life, meeting Sarah in Chicago, wedding at Lake Geneva, career progression, births of children, family losses
Continuous Daily AI Companion	2024–2026 (1,095 days)	4 sessions/day following natural circadian rhythm: Morning (7:30 AM), Midday (12:30 PM), Evening (6:45 PM), Night (10:15 PM)

Corpus Scale

Metric	Value
Total memory records	19,512
Ingested memories (with consolidation)	20,397 (10,254 Semantic + 10,143 Episodic)
Hebbian associative edges	40,792
Temporal chain linked slots	20,397
Calendar days covered	1,095 continuous days + 33 years of biographical depth
Sessions per day	3–4 (realistic human rhythm)

4-Generation Kinship Tree (14 Individuals)

MindSpan includes an explicit, structured kinship hierarchy:

• Generation 1 (Great-Grandparents): Arthur & Clara Thompson (WWII veteran, 1944 Elgin watch, heritage quilt); Salvatore & Rosa Moretti (Ellis Island immigrants, Sunday gravy tradition, antique cameo brooch)
• Generation 2 (Grandparents): Tom & Linda Thompson (retired teacher/librarian); Patricia Moretti-Miller (retired oncology nurse) & late Robert Miller (master cabinetmaker, Lie-Nielsen tools)
• Generation 3 (Mike's Generation): Mike & Sarah; Dr. Emily Thompson-Reed & Mark Reed; Daniel Miller & Chloe Vance; cousins Leo Moretti (brewmaster) & Jessica Reed-Clark (veterinarian)
• Generation 4 (Children): Ethan, Lily, cousin Maya Reed

This enables multi-hop relational reasoning: "What did Mike's father-in-law bequeath to him?" requires traversing Mike → wife Sarah → father Robert Miller → Lie-Nielsen hand planes → walnut desk project.

Neuromorphic Metadata

Every memory record carries biologically-grounded metadata:

Field	Range	Purpose
Valence	-128 to +127	Emotional polarity (grief vs. joy)
Arousal	0 to 255	Physiological intensity
Importance (ICNU)	0.0 to 10.0	Cognitive significance score
Synaptic Tags	String arrays	Semantic clustering (["woodworking", "Robert-Miller", "Lie-Nielsen"])

Dual-Perspective Memory

For every user interaction, there's a corresponding AI observation (bio-XXXX-j) capturing sensory cues, physiological tracking, and semantic consolidation — mirroring how a real cognitive system would encode memories from both first-person and observer perspectives.

---

2. The 17 Cognitive Tracks

MindSpan queries span 17 specialized cognitive reasoning tracks, each designed to stress different aspects of memory architecture:

Track	What It Tests
KINSHIP_HERITAGE	Multi-hop kinship resolution, heirloom provenance
TEMPORAL_ANCHORS	Anchoring events across decades (2009 UIUC → 2013 Chicago → 2022 Frisco)
BIOGRAPHICAL_CHRONOLOGY	Life timeline ordering and epoch transitions
EPISODIC_PRECISION	Exact recall of specific episodes with contextual detail
SOCIAL_DYNAMICS	Interpersonal relationships, workplace mentoring, family dynamics
CAREER_DECISIONS	Professional trajectory, promotion reasoning, technical leadership
ENTERPRISE_ARCHITECTURE	Software architecture decisions, FHIR compliance, clinical feedback
CRAFT_MASTERY	Woodworking techniques, tool provenance, material selection
FAMILY_LIFE	Children's milestones, activities, developmental tracking
HEALTH_WELLNESS	Running check-ins, eldercare medical tracking
FINANCIAL_DECISIONS	529 savings, mortgage decisions, tax planning
LOGISTICAL_GROUNDING	Daily logistics, scheduling, home organization
DAILY_LIVING	Routine activities, meal planning, pet care
HOME_MAINTENANCE	House repair, seasonal maintenance, contractor coordination
ELDERCARE_LOGISTICS	Tom's knee surgery recovery, Patricia's care coordination
EARLY_CHILDHOOD	Retrieval of 20+ year old childhood memories under decay
COMMUNITY_ENGAGEMENT	Neighborhood, volunteering, local events

---

3. Why MindSpan Is Superior to LongMemEval

3.1 LongMemEval: A Forensic Assessment

LongMemEval (UCLA & Tencent AI Lab, ICLR 2025) was a notable first step in evaluating long-term chat memory. However, our forensic analysis reveals critical structural limitations:

Limitation	Evidence
Short temporal horizon	Average target fact age: 16.4 days. 59.7% of facts are ≤7 days old. 88.1% are ≤30 days old. Only 1.4% exceed 6 months.
Artificial session compression	LongMemEval-M crams 475 sessions into 29 calendar days — that's 16 distinct sessions/day. No human interacts this way.
Multi-user scraped distractors	Haystack consists of ShareGPT and UltraChat transcripts from thousands of anonymous users. Distractors are about quantum physics, pasta recipes, and C++ debugging — not plausible within a single person's life.
Flat metadata	Every turn: valence: 0, arousal: 0, importance: 1.0. No emotional differentiation.
No relational structure	Zero kinship, zero social graph. The only entities are "User" and "Assistant."
5 query types only	Information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention.

3.2 Head-to-Head Comparison

Dimension	LongMemEval	MindSpan
Temporal span	~27 calendar days avg (max 304)	20+ years + 1,095 continuous days
Target fact age	16.4 days avg	2 months to 20+ years
Corpus size	10,866 turns (940 sessions)	19,512 records
Sessions/day	16 (unrealistic)	3–4 (circadian-realistic)
Persona coherence	Multi-user scraped transcripts	Single coherent longitudinal persona
Kinship graph	None	4-generation, 14 individuals
Affective metadata	Flat zeros	Full valence/arousal/importance
Memory taxonomy	Flat episodic turns	Episodic + Semantic dual-perspective
Query categories	5	17 cognitive tracks
Cognitive mechanisms	Basic vector/keyword search	Hebbian networks, Ebbinghaus decay, REM consolidation, gated graph traversal
Token budget	Unbounded	Strict <1,800 tokens

3.3 The Core Argument

LongMemEval tests short-term buffer retrieval under artificial noise. When 88% of your target facts are less than a month old and your distractors are unrelated multi-user chat logs, you're measuring keyword filtering — not memory.

MindSpan tests true lifelong cognitive memory. When the system must retrieve a 20-year-old childhood memory about shag carpet and dial-up modems, disambiguate between Ethan's soccer practice in May 2024 vs. September 2025, or traverse a kinship tree to identify that Robert Miller's Lie-Nielsen No. 7 jointer plane was inherited through marriage — that is cognitive memory retrieval.

---

4. Benchmark Results

Spector Memory on MindSpan (natural-ingest-v1)

Metric	Value
QA Accuracy	200/200 (100.0%)
Model	gemini-3.1-flash-lite
Cognitive NDCG@10	0.5344 (+32.9% over dense baseline)
Similarity NDCG@10	0.6128
Baseline NDCG@10	0.2051
Wins / Ties / Losses	115 / 61 / 24
Avg retrieval tokens	1,731 (strict ceiling: 1,800)

Every single one of the 17 cognitive tracks achieves 100% QA accuracy.

Spector Memory on LongMemEval (for reference)

Metric	Value
QA Accuracy	64.40% (322/500)
Pure Recall Latency	20 ms
End-to-End Latency	731 ms

Note: LongMemEval's 64.4% reflects the dataset's own structural issues (scraped distractor noise, flat metadata) more than Spector's capabilities. Industry baselines on LongMemEval: Zep reports 75–80%, Mem0 reports 62–68%.

---

5. Dataset Availability

The MindSpan benchmark dataset is located in the spector-datasets repository under mindspan/. Key files:

File	Description
data/corpus.jsonl	19,512 memory records with full metadata
data/queries.jsonl	200 benchmark queries across 17 tracks
data/qrels.tsv	Gold relevance judgments
data/persona.json	Mike Thompson persona definition
data/kinship_tree.json	4-generation family tree (14 individuals)
results/natural-ingest-v1/summary.json	Full benchmark results

---

6. What's Next

• Expanding to the full 500-query evaluation suite
• Publishing the MindSpan specification and benchmark methodology paper
• Adding Ebbinghaus decay stress tracks and counterfactual suppression queries
• Community benchmark submissions welcome

We believe MindSpan establishes a new gold standard for evaluating lifelong cognitive memory systems. We welcome discussion, feedback, and contributions from the community.