Context compaction for long sessions

[badlogic]

Overview

Context compaction summarizes conversation history when approaching context limits, allowing long sessions to continue without hitting the wall.

See packages/coding-agent/docs/compaction.md for research on how Claude Code, Codex CLI, OpenCode, and Amp handle this.

Commands

• /compact [custom instructions] - Manual compaction trigger. Optional custom instructions let users guide what to focus on in the summary.
• /autocompact - Opens selector UI to toggle auto-compaction on/off. Also displays current power-user settings (reserveTokens, keepRecentTokens).

Configuration

Settings stored in ~/.pi/agent/settings.json:

interface Settings {
  // ... existing fields
  compaction?: {
    enabled?: boolean           // default: true, toggled via /autocompact
    reserveTokens?: number      // default: 16384, power-user setting
    keepRecentTokens?: number   // default: 20000, power-user setting
  }
}

Why these defaults:

• reserveTokens: 16384 - Room for summary output (~13k) plus safety margin (~3k)
• keepRecentTokens: 20000 - Preserves recent context verbatim, summary focuses on older content

Token Calculation

Context tokens are calculated from the last non-aborted assistant message using the same formula as the footer:

contextTokens = usage.input + usage.output + usage.cacheRead + usage.cacheWrite

This gives total context size across all providers. The input field represents non-cached input tokens, so adding cacheRead and cacheWrite gives the true total input.

Trigger condition:

if (contextTokens > model.contextWindow - settings.compaction.reserveTokens) {
  await compact({ auto: true });
}

Turn Boundaries

Messages follow patterns like: user, assistant, toolResult, toolResult, user, assistant, ...

Critical rule: Never cut mid-turn. A turn = user message → assistant responses + tool results until next user message. Always cut before a user message to keep assistant + toolResult pairs intact (providers fail if toolResult is orphaned from its assistant message with the toolCall).

Summary Injection

The summary is injected as a user message with a prefix (similar to Codex approach). This makes it visible to the user and clearly frames it for the model.

Prefix:

Another language model worked on this task and produced a summary. Use this to continue the work without duplicating effort:

Session File Format

Compaction events are appended to the session file (never inserted mid-file):

interface CompactionEvent {
  type: "compaction"
  timestamp: string
  summary: string           // The summary text
  keepLastMessages: number  // How many messages before this event to keep
  tokensBefore: number      // Context size before compaction
}

Example: Single Compaction

Session file with messages (u=user, a=assistant, t=toolResult):

u1, a1, t1, t1, a1, u2, a2, u3, a3, t3, a3, t3, a3, u4, a4, t4, a4

Compaction triggers, keeping last 4 messages. The compaction event is appended:

u1, a1, t1, t1, a1, u2, a2, u3, a3, t3, a3, t3, a3, u4, a4, t4, a4
[COMPACTION: summary="...", keepLastMessages=4]

Session loader builds context:

[summary_as_user_msg], u4, a4, t4, a4

New messages after compaction are appended:

u1, a1, t1, t1, a1, u2, a2, u3, a3, t3, a3, t3, a3, u4, a4, t4, a4
[COMPACTION: summary="...", keepLastMessages=4]
u5, a5

Session loader now builds:

[summary_as_user_msg], u4, a4, t4, a4, u5, a5

Example: Multiple Compactions

After more messages, second compaction triggers:

u1, a1, t1, t1, a1, u2, a2, u3, a3, t3, a3, t3, a3, u4, a4, t4, a4
[COMPACTION 1: summary="...", keepLastMessages=4]
u5, a5, u6, a6, t6, a6, u7, a7
[COMPACTION 2: summary="...", keepLastMessages=3]

Session loader finds COMPACTION 2 (latest), builds:

[summary2_as_user_msg], u6, a6, t6, a6, u7, a7

Note: COMPACTION 2's summary incorporates COMPACTION 1's summary because the summarization model received the full current context (which included summary1 as first message).

Boundary rule: When calculating keepLastMessages for COMPACTION 2, we only count messages between COMPACTION 1 and COMPACTION 2. If keepLastMessages exceeds the available messages (e.g., keepLastMessages=10 but only 6 messages exist after COMPACTION 1), we take all available messages up to the boundary. We never cross a compaction boundary.

Summarization

Use pi-ai directly (not the full agent loop) for summarization:

• No tools needed
• Set maxTokens to 0.8 * reserveTokens (leaves 20% for prompt overhead and safety margin)
• Pass abort signal for cancellation
• Use the currently selected model
• Reasoning disabled (thinking level "off") since we just need a summary, not extended reasoning

With default reserveTokens: 16384, maxTokens = ~13107.

Prompt (based on Codex, enhanced):

You are performing a CONTEXT CHECKPOINT COMPACTION. Create a handoff summary for another LLM that will resume the task.

Include:
- Current progress and key decisions made
- Important context, constraints, or user preferences
- Absolute file paths of any relevant files that were read or modified
- What remains to be done (clear next steps)
- Any critical data, examples, or references needed to continue

Be concise, structured, and focused on helping the next LLM seamlessly continue the work.

Auto-Compaction Trigger

Auto-compaction is checked in the agent subscription callback after each message_end event for assistant messages. If context tokens exceed the threshold, compaction runs.

Why abort mid-turn: If auto-compaction triggers after an assistant message that contains tool calls, we abort immediately rather than waiting for tool results. Waiting would risk:

1. Tool results filling remaining context, leaving no room for the summary
2. Context overflow before the next check point (agent_end)

The abort causes some work loss, but the summary captures progress up to that point.

Trigger flow (similar to /clear command):

async handleAutoCompaction(): Promise<void> {
  // 1. Unsubscribe to stop processing events (no more messages added to state/session)
  this.unsubscribe?.();
  
  // 2. Abort current agent run and wait for completion
  this.agent.abort();
  await this.agent.waitForIdle();
  
  // 3. Stop loading animation
  if (this.loadingAnimation) {
    this.loadingAnimation.stop();
    this.loadingAnimation = null;
  }
  this.statusContainer.clear();
  
  // 4. Perform compaction on current state:
  //    - Generate summary using pi-ai directly (no tools, reasoning off)
  //    - Write compaction event to session file
  //    - Rebuild agent messages (summary as user msg + kept messages)
  //    - Rebuild UI to reflect new state
  
  // 5. Resubscribe to agent
  this.subscribeToAgent();
  
  // 6. Show compaction notification to user
}

This mirrors the /clear command pattern: unsubscribe first to prevent processing abort events, then abort and wait, then do the work, then resubscribe.

Error Handling

• On compaction failure: output error, let user decide what to do
• In JSON/RPC mode: emit {"type": "error", "error": "message"} (existing pattern)
• Compaction is abortable via the same abort signal as regular streaming

Image Handling

Two cases:

1. Images via file path in prompt → Model reads with tool → Can be captured in summary as "image at /path/to/file.png was analyzed". Prompt instructs model to include absolute file paths.
2. Images via @attachment → Attached to user message directly → Lost in compaction (cannot summarize an image). Known limitation.

Interaction with /branch

The /branch command lets users create a new session from a previous user message. With compaction:

• Branch UI reads from session file directly (not from state.messages) to show ALL user messages, including those before compaction events
• Branching copies the raw session file line-by-line up to (but excluding) the selected user message, preserving all compaction events and intermediate entries

Why read from session file instead of state.messages

After compaction, state.messages only contains [summary_user_msg, ...kept_messages, ...new_messages]. The pre-compaction messages are not in state. To allow branching to any historical point, we must read the session file directly.

Reworked createBranchedSession

Current implementation iterates state.messages and writes fresh entries. New implementation:

1. Read session file line by line
2. For each line, check if it's the target user message
3. Copy all lines up to (but excluding) the target user message
4. The target user message text goes into the editor

Example: Branching After Compaction

Session file:

u1, a1, u2, a2
[COMPACTION: summary="...", keepLastMessages=2]
u3, a3, u4, a4

User branches at u3. New session file:

u1, a1, u2, a2
[COMPACTION: summary="...", keepLastMessages=2]

Session loader builds context for new session:

[summary_as_user_msg], u2, a2

User's editor contains u3's text for editing/resubmission.

Example: Branching Before Compaction

Same session file, user branches at u2. New session file:

u1, a1

No compaction in new session. Session loader builds:

u1, a1

This effectively "undoes" the compaction, letting users recover if important context was lost.

Modes

Works in all modes:

• TUI: Commands available, UI shows compaction happening
• Print/JSON: Compaction events emitted as output
• RPC: Compaction events sent to client

Implementation Steps

1. Add compaction field to Settings interface and SettingsManager
2. Add CompactionEvent type to session manager
3. Update session loader to handle compaction events (find latest, apply keepLastMessages with boundary rule)
4. Rework createBranchedSession to copy raw session file lines instead of re-serializing from state
5. Update /branch UI to read user messages from session file directly
6. Add /compact command handler
7. Add /autocompact command with selector UI
8. Add auto-compaction check in subscription callback after assistant message_end
9. Implement handleAutoCompaction() following the unsubscribe/abort/wait/compact/resubscribe pattern
10. Implement summarization function using pi-ai (no tools, reasoning off)
11. Add compaction event to RPC/JSON output types
12. Update footer to show when auto-compact is disabled

[badlogic]

Progress Update

Done

Session Manager (session-manager.ts):

• CompactionEntry type with firstKeptEntryIndex (absolute index into entries)
• SessionEntry union type for all entry types
• LoadedSession interface
• parseSessionEntries() - parse JSONL content
• createSummaryMessage() - create user message with summary prefix
• loadSessionFromEntries() - reconstruct messages handling compaction events
• SessionManager.loadSession() - unified loading with compaction support
• SessionManager.loadEntries() - raw entry loading
• SessionManager.saveCompaction() - append compaction entry

Compaction Logic (compaction.ts):

• CompactionSettings type and defaults
• calculateContextTokens() - sum usage tokens
• getLastAssistantUsage() - find last valid usage from entries
• shouldCompact() - threshold check
• findCutPoint() - find entry index respecting turn boundaries and token budget
• generateSummary() - LLM call for summarization
• compact() - main function, returns CompactionEntry to append

Tests (compaction.test.ts):

• Unit tests for all functions
• Integration tests with real session fixture (~1000 entries)
• LLM integration tests (with ANTHROPIC_OAUTH_TOKEN)

Still TODO

1. Settings - Add compaction field to Settings interface in settings-manager.ts

2. Commands:

   • /compact [custom instructions] - manual trigger
   • /autocompact - toggle with selector UI

3. Auto-compaction trigger in TUI:

   • Check after message_end events
   • handleAutoCompaction() following unsubscribe/abort/wait/compact/reload/resubscribe pattern

4. Footer - Show when auto-compact is disabled

5. RPC/JSON modes - Emit compaction events

6. /branch rework - Read from session file directly to show all historical user messages

[badlogic]

Implementation Complete

All items from the issue are now implemented:

Done

• Settings: Added compaction field to Settings with enabled, reserveTokens, keepRecentTokens
• /compact [instructions]: Manual compaction with optional custom focus
• /autocompact: Toggle auto-compaction on/off
• Auto-compaction trigger: Checks after each assistant message_end, triggers when threshold exceeded
• Footer: Shows (auto) after context percentage when auto-compact is enabled
• RPC mode: Added {type: "compact", customInstructions?} command, emits CompactionEntry on success
• /branch rework: Now reads from session file directly to show ALL historical user messages (including pre-compaction)
• createBranchedSessionFromEntries: Copies raw session entries preserving compaction events

Testing

• All 21 compaction tests pass
• LLM integration tests work with real session fixture

[badlogic]

Implemented in v0.12.7.

Documentation:

• Context Compaction (README)
• RPC Protocol - Compaction