Session lineage model: fork/resume/compaction content duplication corrupts counts, cost, search

[Sinity]

Problem

Polylogue ingests every physical session artifact as an independent session with
its own full message tree. But agentic tooling routinely produces multiple
physical artifacts that share content — resumed Codex threads replay their
parent, parallel agent fan-outs fork one context into N rollouts, Claude
auto-compaction re-emits a whole conversation to summarize it. The shared content
is stored and counted once per physical artifact, so the archive misrepresents
its own contents.

The lineage is already detected (session_profiles.logical_session_id groups
fork families correctly). The failure is that storage and nearly every
aggregate ignore it.

Blast radius (measured on the live 37 GB archive)

• Sessions: 16,414 physical vs 8,807 logical roots (1.86×).
• Messages: ~32 % duplicate-bearing (Codex fork replays, 1.77M rows).
• Block text: 42 % non-unique (includes legitimate recurrence + fork replay).
• Cost/tokens: Codex 189.7 B physical vs 139.3 B authoritative state_5.sqlite
  (~50 B / 1.36× over-count, entirely inside fork families).
• FTS indexes duplicated content → duplicate hits, skewed ranking.
• Embeddings (when enabled) would pay to embed the same content many times.

Root causes (code-confirmed)

1. No cross-session content dedup at ingest (pipeline/ids.py,
   storage/sqlite/archive_tiers/write.py). content_hash is per-session, so
   identical text across forks hashes differently and cannot dedupe.
2. Aggregates count physical sessions: cost_compute, session_model_usage,
   stats.get_stats_by, aggregate_message_stats, list_summaries,
   fts5/hybrid, embeddings, cost rollups. Only session_tag_rollups,
   get_logical_session, and ThreadInsight are logical-aware.
3. agent-acompact-* misclassified as subagent (claude/code_parser.py
   keys on fallback_id.startswith("agent-")). Auto-compaction is a derived
   re-summarization of the parent, not an independent session (187 sessions).

Not a bug

Real Task subagents (7,383) are legitimate distinct work (~99 % unique
content) and must keep counting. The duplication is narrow: Codex resume+fork
families and Claude auto-compaction.

Design

Full taxonomy, correct model (content-addressed blocks; logical session = union
DAG), and a phased plan in docs/design/session-lineage-model.md:

• Phase 0 — fix agent-acompact-* misclassification (small).
• Phase 1 — make aggregates lineage/content-aware at read time (no schema
  change; fixes the numbers; validated against state_5.sqlite / stats-cache.json).
• Phase 2 — structural content-addressing so shared content is stored once
  (schema change + re-ingest; makes the bug impossible by construction).

Open decisions

1. Default reporting unit: logical (recommended) vs physical headline counts/cost.
2. Whether Phase 2 structural overhaul is in scope now.
3. Re-ingest tolerance for Phases 0/2.

Ref design doc; discovered while reconciling cost accounting (the cached/disjoint-lane
and pricing-catalog fixes landed separately on feature/operator-dogfood-hardening).

[Sinity]

Updated the design after reading the Codex/Claude sources and verifying all forms. Key correction: dedup is by lineage, not content-addressing — content-addressing would wrongly merge coincidentally-identical unrelated messages.

Verified model (now in docs/design/session-lineage-model.md): two structural concepts, each unified across providers —

1. Prefix inheritance (fork/resume): Codex denormalizes (copies full parent history into the child rollout via thread_spawn/forked_from_id); Claude threads via uuid/parentUuid (no copy). We store the child's own messages + a lineage edge; never the copy.
2. Context discontinuity (compaction): Codex compacted+replacement_history; Claude inline isCompactSummary; Claude agent-acompact-* = compaction implemented as a 100%-copy (187 sessions, ~2 unique blocks each). We model a boundary + summary-as-real-message; pre-compaction messages kept once.

Fresh Task subagents (agent-<hex>, ~99% unique) are real work, stored whole. Decisions: logical default, normalize copies (not store-and-flag), re-ingest acceptable.