Session forking writes full parent rollout into child JSONL (storage amplification despite forked_from_id)

[Yasei-no-otoko]

What version

• openai/codex (desktop + CLI)

Severity

P1 - local disk growth and resumability risk from persistent thread fork storage amplification

What is happening

A newly forked/branched thread can still persist a full parent-history payload instead of only fork-local delta when forked_from_id exists, producing large duplicated .jsonl rollouts and rapidly exhausting disk.

Observed local reproduction matches the following parent/thread IDs:

• Parent thread: 019e1823-6e48-75f0-a644-c8c043cef937
• Child thread: 019f9ff6-d5eb-7292-9446-add44e4024b2
• Child thread: 019fa108-23a7-7921-ab39-0092e1227f2d

These files are in ~/.codex/sessions/2026/07/27/ and are still showing duplicated large inherited content characteristics unless rewritten, and they are expected to be reduced to local-only delta with a parent cutoff (history_base).

Expected behavior

• Child thread rollouts must not materialize parent history inline when a fork boundary exists.
• Child rollouts should store the minimum delta after a valid forked_from_id / boundary offset.
• Legacy readers must remain compatible, with explicit history_base metadata for managed children.

Why this is urgent

• Existing local report already shows large local growth and high-pressure behavior in ~/.codex/sessions.
• Parent-child duplication can silently multiply costs for long sessions and repeated forking.
• It also increases startup/replay and context-reconstruction cost.

Related context

I have a local patch-set with hardening around:

• managed child validation at read/list/fork paths
• path/rollout lock scoping fixes to avoid resume deadlock
• reference child rejection for malformed / external rows
• fork/reference history cutoff handling in thread-store and rollout_lineage

Related issues already discussed:

• 22593 (Session forks duplicate parent history on disk)
• 34337 (CLI + Desktop session rollouts consume huge disk)
• 34268 (forks duplicate compaction snapshots)

Please triage as urgent because this is a repeatable storage amplification class and affects both CLI/Desktop shared persistence.

[github-actions]

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• Codex Desktop: screenshots re-persisted in full on every compaction and inherited by subagent forks - ~/.codex/sessions reached ~165 GiB (95% base64 images) #35458

Powered by Codex Action

[Yasei-no-otoko]

Implementation update (2026-07-28):

• Full open-PR audit: 193 open PRs checked; no existing PR implements managed history_base references plus fork-local JSONL deltas. [core] Drop parent MCP lifecycle events from forked agent history #30977 is related payload filtering but does not change child persistence.
• Fix branch: Yasei-no-otoko:session-jsonl-amplification-upstream
• Head: 61cb3d7565bf6409ea10bd55082675797086085b
• Updated onto upstream main at 49025589b0216b876b1a6a20977536c7d55cdb8b via merge commit 477a26d4d2.
• Validation: cargo +1.95.0 test -p codex-thread-store --lib => 192 passed, 0 failed; package fmt check passed.
• Added regression coverage for concurrent archive versus legacy fork mode dispatch and Paginated Latest cutoff after blank/rejected complete physical tail lines.
• The two reported local child rollouts are now delta-only references: 67,930 bytes / 16 records and 23,463 bytes / 2 records, each with a frozen parent history_base cutoff.

Compare URL: https://github.com/openai/codex/compare/main...Yasei-no-otoko:codex:session-jsonl-amplification-upstream?expand=1

I cannot open the upstream PR from the current account: GitHub returns Yasei-no-otoko does not have the correct permissions to execute CreatePullRequest because this repository's external contribution flow is invitation-only. The branch is pushed and ready for an invited maintainer/account to open the PR. The PR should use Fixes #35647 and Related to #22593.

[Yasei-no-otoko]

Additional reproduction details for Issue 1 have been confirmed.

The affected child’s legacy history_base.end_byte_offset falls immediately after a newline in the parent JSONL, so it is physically at a complete record boundary. However, the preceding record’s token_count.rate_limits used an older or newer schema that differs from the current Rust type. As a result, legacy_fork::last_complete_jsonl_offset_at_or_before treated a nested RolloutLine deserialization failure as incomplete JSONL, causing resume to stop with:

invalid rollout history lineage for 019fa108-23a7-7921-ab39-0092e1227f2d: cutoff byte offset is not at a complete JSONL record

The fix restricts cutoff validation to the stable outer rollout envelope (timestamp: string, type: string, payload: object) and permits evolution of nested payload schemas. It requires the rollout-envelope shape rather than accepting arbitrary objects, and continues to reject malformed envelopes and payload: null or array values.

The regression tests and all 195 codex-thread-store tests pass.

Fix PR (submitted from a fork): Yasei-no-otoko#5

[MengShengbo]

I reproduced this on Codex CLI 0.146.0 on Linux x86_64 and have a second, sanitized data point that narrows the high-frequency trigger.

Observed chain

A single long-lived process launched as plain codex created seven top-level user-thread children over 54 minutes. Their rollout sizes were:

62.68 MiB
63.56 MiB
63.80 MiB
63.87 MiB
63.98 MiB
64.02 MiB
65.08 MiB

All seven canonical session_meta records reported:

source = cli
originator = codex-tui
thread_source = user
history_mode = legacy
cli_version = 0.146.0

Each child had a fresh thread ID and forked_from_id pointing to the immediately preceding thread, producing one continuous lineage. This is about 447 MiB of child rollouts from one active conversation in under an hour.

Trigger classification

This chain is not byte-identical duplicate files and does not look like same-ID resume corruption. It is source-preserving prompt-edit/turn-replacement branching:

• Several source turns ended with the durable <turn_aborted> marker.
• For four corrected prompts, the discarded prompt occurred twice in the source rollout (response_item + event_msg) and zero times in the child's inherited initial batch.
• The replacement child retained the earlier conversation and excluded the interrupted prompt, which is consistent with a branch-before-turn operation.
• Model and effort remained gpt-5.5 / medium; I found no persisted model-safety-buffering event in the chain, so this does not match the faster-model safety retry path.

The branch behavior itself appears intentional. The storage amplification is not: prompt editing is a normal, potentially frequent TUI workflow, and every edit materialized nearly the complete legacy parent history again.

Persistence evidence

On each child creation, inherited rollout items were serialized again with timestamps clustered within roughly one second of the child's creation time. Historical session_meta records were inherited too, so the number of persisted session_meta lines increased by one at each generation. This explains why exact-file hashing finds no duplicates even though almost all logical history is copied.

The current implementation trace matches the analysis already posted here and in #22593:

• legacy root/user forks reach ForkPersistence::Copied;
• record_initial_history persists the reconstructed inherited items into the child rollout;
• paginated forks already have the reference-backed history_base path from Support paginated thread forks #35220, but existing and newly continued legacy threads remain on copied persistence.

I am also aware of the external patch at Yasei-no-otoko#5 and do not want to create a competing unsolicited implementation.

Contribution request

The repository contribution guide requires an explicit maintainer invitation before opening an external PR. If the team wants an external contribution here, I can prepare a focused, test-first PR against the preferred architecture rather than submitting another broad lineage rewrite.

A possible first reviewable stage would be a regression test for a legacy TUI prompt-edit fork showing that the child persists a bounded parent reference/delta rather than reserializing the complete parent rollout. Please confirm whether the intended direction is:

1. reference-backed legacy forks,
2. migration/new-thread defaulting to paginated history, or
3. another staged approach already planned internally.

I can provide additional aggregate-only measurements without exposing conversation contents.