Paginated history drops valid flattened rollout records and reuses ordinals

[Tsury]

What version of Codex CLI is running?

Observed on 0.146.0-alpha.10.1. The affected source paths remain unchanged in rust-v0.146.0-alpha.14.

What platform is your computer?

Linux x86_64.

What issue are you seeing?

Paginated rollout history has inconsistent RolloutLine decoding. Some readers deserialize JSON directly into RolloutLine, while the canonical loader first parses into serde_json::Value and then calls serde_json::from_value::<RolloutLine>.

A serialized RolloutLine::EventMsg(EventMsg::TokenCount(...)) containing populated rate-limit and credit data is valid JSON and succeeds through the value-first path, but direct serde_json::from_str::<RolloutLine> / serde_json::from_slice::<RolloutLine> rejects it.

Two paginated-history readers currently use the direct path:

• codex-rs/rollout/src/ordinal.rs: scan_next::<RolloutLine>()
• codex-rs/thread-store/src/local/thread_history_materialization.rs: serde_json::from_slice(line_bytes)

This produces two related failures:

1. Resume ordinal discovery silently skips the valid final record, steps back to an earlier ordinal, and can append a new record with an already-used ordinal.
2. SQLite history projection silently skips the valid record while advancing its byte offset beyond it, leaving the materialized projection behind the canonical JSONL permanently.

Once affected, catch-up can begin with one replayed boundary ordinal equal to projection_state.next_ordinal - 1; rejecting that boundary prevents the remaining valid suffix from materializing.

What steps can reproduce the bug?

Add a focused test that serializes a paginated RolloutLine containing a token-count event with rate-limit and credit fields:

let encoded = serde_json::to_string(&RolloutLine {
    timestamp: "2026-07-09T00:00:05Z".to_string(),
    ordinal: Some(5),
    item: RolloutItem::EventMsg(EventMsg::TokenCount(TokenCountEvent {
        info: Some(TokenUsageInfo::full_context_window(1_000)),
        rate_limits: Some(RateLimitSnapshot {
            limit_id: Some("limit-1".to_string()),
            limit_name: None,
            primary: Some(RateLimitWindow {
                used_percent: 1.0,
                window_minutes: Some(60),
                resets_at: Some(1),
            }),
            secondary: None,
            credits: Some(CreditsSnapshot {
                has_credits: true,
                unlimited: false,
                balance: Some("1".to_string()),
            }),
            individual_limit: None,
            spend_control_reached: None,
            plan_type: None,
            rate_limit_reached_type: None,
        }),
    })),
})?;

assert!(serde_json::from_str::<RolloutLine>(&encoded).is_err());
assert!(
    serde_json::from_str(&encoded)
        .and_then(serde_json::from_value::<RolloutLine>)
        .is_ok()
);

Then:

1. Append that record as the final line of a paginated rollout and resume it. Ordinal discovery ignores ordinal 5.
2. Materialize the same rollout into the thread-history SQLite projection. The line is rejected, but the stored byte offset advances past it.

What is the expected behavior?

Every rollout reader should use the same value-first decoding semantics as the canonical loader. Resume should continue after the real final ordinal, and SQLite materialization should project every valid canonical record.

For projections already affected by the old decoder, narrowly accepting and skipping one replayed first boundary ordinal (next_ordinal - 1) allows the remaining suffix to catch up without rewriting canonical JSONL.

Additional information

rust-v0.146.0-alpha.14 changes nearby thread-list behavior for missing rollout paths, but ordinal.rs and thread_history_materialization.rs still use direct RolloutLine deserialization.

[erichanwang]

i traced this to the two direct RolloutLine decode sites called out here: reverse ordinal discovery and thread-history materialization both bypass the value-first decode path used by canonical rollout loading.

local tested fix: make those readers parse the physical JSON line as serde_json::Value first, then decode the RolloutLine, preserving the existing behavior of skipping malformed physical lines.

validation on my branch:

• just fix -p codex-rollout
• just fix -p codex-thread-store
• just test -p codex-rollout (113 passed)
• just test -p codex-thread-store (158 passed)

because docs/contributing.md says external PRs are invitation-only, i did not open a PR. branch is ready on my fork if maintainers want one: https://github.com/erichanwang/codex/tree/fix/paginated-rollout-value-decode

[AlarmingMelon]

Confirmed on ChatGPT/Codex for Windows.

Feedback ID: 019fb9de-3796-71e2-b216-b709023f8dbe
Codex app: 26.727.6591.0
CLI/app-server: 0.146.0-alpha.9.2

The raw rollout remains valid with 2,200+ continuous ordinals, but the UI history projection is frozen immediately before ordinal 20. Logs repeatedly report:

invalid type: map, expected f64
thread history projection ... expected ordinal 20, got 21

Impact:

• Reopening the task lost the visible history twice.
• The UI showed only one user and one assistant message.
• Two attempts to branch from a later turn failed.
• Restarting Codex did not repair the projection.

Current-session diagnostics were included with the feedback submission; browser-tab logs were excluded.

[AlarmingMelon]

Two additional Codex for Windows tasks now reproduce the same projection failure on the same installed build:

• App: 26.727.6591.0
• CLI/app-server: 0.146.0-alpha.9.2

Additional affected task 1

• Thread/Feedback ID: 019fc3d5-e9dd-7f51-be4d-8557ffb5a2bd
• At inspection time, the canonical rollout contained 155 valid JSONL records with continuous ordinals 0-154, no malformed JSON, and no gaps or duplicate ordinals.
• SQLite thread_history_projection_state remained frozen at byte offset 235257, next ordinal 17, with only four projected items.
• Logs repeatedly report:
  • invalid type: map, expected f64
  • thread history projection ... expected ordinal 17, got 18

Additional affected task 2

• Thread/Feedback ID: 019fbf98-130c-7e62-b16e-038a76c60401
• The canonical rollout contains 432 valid JSONL records with continuous ordinals 0-431, no malformed JSON, and no gaps or duplicate ordinals.
• SQLite projection is frozen at byte offset 739950, next ordinal 17, again with only four projected items.
• Logs repeatedly report the same map/f64 decoding error followed by expected ordinal 17, got 18.

For both tasks, the app-level read-only history call also fails with:

paginated threads do not support thread/read(includeTurns=true)

User-visible result: reopening either task displays only the initial user input and one assistant response despite the intact canonical rollout.

Negative control: in the same app session, the user reopened two other threads under another project and both appeared to load normally. This suggests the defect is not triggered visibly for every task, even on the same client installation.

Impact now extends beyond missing UI history: substantial weekly model usage is being consumed diagnosing the client and reconstructing work that should remain visible.

Please ensure the eventual repair handles already-stalled projections and the replayed boundary ordinal, rather than only correcting decoding for newly created projections.

[AlarmingMelon]

A fourth affected task reproduces the same decoder/projection failure and adds a potentially important cross-device trigger.

• Thread/Feedback ID: 019fc4f1-b75b-70c0-a8d5-c02fbc9c0003
• Feedback has been submitted with current session logs.
• Windows host app: 26.727.6591.0
• CLI/app-server: 0.146.0-alpha.9.2

Cross-device sequence

1. The task was actively running on the Windows host, where the complete conversation remained visible in the originating client.
2. At approximately 8:20 PM ET, the same active task was opened remotely in Codex on a MacBook.
3. The MacBook displayed the original input and the currently streaming reasoning/thoughts, but none of the completed intervening turns.
4. The Windows client continued showing the complete in-memory conversation until the user navigated away from that task.

Read-only diagnostics on the Windows host

• Canonical rollout: 165 valid JSONL records
• Malformed JSON: 0
• SQLite thread_history_projection_state: byte offset 439027, next ordinal 17
• Only four projected items, at ordinals 9, 10, 12, and 15
• Ordinal 17 is a token_count event whose rate_limits value is the structured map already implicated in this issue
• Logs report:
  • invalid type: map, expected f64
  • thread history projection ... expected ordinal 17, got 18
• The app-level history call fails with:
  • paginated threads do not support thread/read(includeTurns=true)

The first matching projection errors were logged at approximately 8:20:16 PM ET, closely matching the remote-open event. This suggests that cross-device viewing of an active task may initiate the paginated history materialization path and expose the decoder defect, while the originating client temporarily retains a complete live/in-memory view.

The remote view did not corrupt the canonical rollout, but once the originating client navigates away, its complete in-memory representation is expected to be replaced by the same stalled projection. In practical terms, the raw JSONL being recoverable does not eliminate the user impact: reconstructing the missing turns, ordering, tool outputs, and conclusions still consumes substantial time and model usage.