Session Compression

Every request Coyote sends to a model carries the whole session history. Sessions grow, models have a fixed input size, and long sessions would eventually stop fitting. Session compression is how Coyote keeps a long session usable: it replaces older messages with a summary and keeps the most recent exchanges word-for-word.

This page explains when compression runs, what it keeps, how the summary is built, how it interacts with long tool loops, and how to tune it.

What compression is and why it exists

Models measure text in tokens (roughly three quarters of an English word each). Every model has a maximum number of input tokens it will accept per request. This page calls that maximum the window; in the model catalog it is the max_input_tokens field.

A session that keeps growing will eventually exceed the window and the provider will reject the request. Even before that point, size costs money: the full history is re-sent on every request, so a 150,000-token session costs 150,000 input tokens per turn.

Compression fixes both problems. Coyote picks a point in the history, asks a model to summarize everything before that point, and replaces those messages with the summary. Messages after that point stay exactly as they were. The folded messages are archived inside the session file, not deleted.

Note: Compression is not free. Providers cache the unchanged prefix of a prompt between requests, and a compression rewrites that prefix from the first message onward, so the next request pays full price for the whole prompt. Compressing too often is itself a cost. The tuning guide below is mostly about finding the balance.

The two triggers

Coyote checks whether a session needs compressing at two moments: just before every request to the model, and again when a turn ends (after the model's final text reply is saved). Two independent triggers are evaluated.

The threshold

compression_threshold is a token count you choose. Coyote keeps a local estimate of the session's size; once that estimate is above the threshold, compression is wanted.

The threshold trigger also requires that compressing would actually reclaim something worthwhile: at least one fifth of the threshold. Without this rule, a session whose system prompt plus kept tail sits just above the threshold would re-compress every turn and gain nothing each time.

The safety valve

compression_safety_valve (default 0.8) is a fraction of the model window. Multiplying the two gives the valve limit: with a 200,000-token window and the default valve, the limit is 160,000 tokens. The valve is a backstop that fires even when the threshold is set too high, or not set usefully at all.

The local token estimate can run low compared with what the provider counts, so the valve compares the limit against the larger of two numbers: Coyote's local estimate, and the prompt-token count the provider reported for the most recent request. The valve trigger only requires that there is something to fold; it has no minimum-reclaim rule.

Both triggers are skipped while a compression is already running, and neither fires when the kept tail would be the whole session (there would be nothing to fold).

Trigger	Fires when	Extra condition
compression_threshold	local estimate > threshold	reclaimable tokens >= threshold / 5
compression_safety_valve	max(local estimate, last reported prompt tokens) >= valve x window	reclaimable tokens > 0

What is kept and what is folded

When compression runs, the session is split in two. The tail is the run of most recent messages that is kept word-for-word. Everything before the tail (apart from the leading system message) is folded into the summary.

The tail is chosen in three steps:

1. Start at the last compression_keep_last messages.
2. Move the start forward to the next user message, so a kept exchange always begins with the user's prompt and is never cut in the middle of a model reply or a tool exchange.
3. While the tail is larger than the token budget, drop its oldest exchange (move the start forward to the next user message again).

So compression_keep_last is an upper bound on how many messages survive, not a guarantee. 0 means keep nothing.

The token budget

The token budget is the smaller of two values:

• compression_threshold / 2, so the session does not immediately re-trigger the threshold after compressing.
• (compression_safety_valve x window) / 4, so the kept tail can never on its own push the next request over the valve limit.

If the threshold is 0 and the model declares no window (or the valve is 0), there is no budget and the tail is limited only by compression_keep_last.

How folding works

The messages before the tail are summarized by the summarizer model: compression_model if you set one, otherwise the session's own model. Because the folded history may be larger than the summarizer's window, it is cut into chunks that each fit about 60% of that window. Each chunk is sent to the summarizer together with the summary of the previous chunk and your summarization_prompt, so the summary rolls forward through the history. If the summarizer model declares no window in the model catalog, the session model's window is used instead; if neither declares one, Coyote assumes 200,000 tokens.

Chunks are cut at message boundaries. A single message too large for one chunk is split further: a tool message at tool-round boundaries, then per tool result, and a single result that is still too big is truncated for the summarizer only; a long text message is split into parts. The saved session file is never truncated.

The result replaces the folded messages with one leading system message that contains the original system prompt followed by the summary, introduced by summary_context_prompt. The tail follows it unchanged.

If the whole session was folded (the tail was empty), the most recent user exchange is re-attached to the next request so the model still sees the last thing you said, but only if that exchange fits the same token budget. Otherwise the model continues from the summary alone.

Tool loops and checkpoints

While the model is calling tools, each tool result is appended to the request in memory and sent back to the model. None of that is part of the session yet; it is written to the session only when the turn ends with a text reply. The triggers above look at the session, so during a long tool loop they cannot see the growing pile of results.

The checkpoint closes that gap. After each model reply that contains tool calls, Coyote measures the pile of in-flight tool activity. If it is at least a quarter of the model window (and at least 2,000 tokens), the pile is committed to the session right then as a finished turn made of three messages. (A model that declares no window is treated as having a 200,000-token window for this check.)

1. Your original prompt.
2. The tool activity so far.
3. A short assistant note: [Context checkpoint: the tool activity above was committed to session history mid-turn so it can be compressed. The task is not finished; it continues in the next turn.]

The loop then continues with an internal prompt that tells the model to carry on exactly where it left off and not to restart or re-summarize. Because the next request starts with the normal pre-request check, the committed tool message is now visible to both triggers and can be folded.

A second checkpoint trigger commits the pile when the whole in-progress request (session plus pile, or the provider's last reported count if larger) has already reached the valve limit. This trigger is latched: once it fires it stays quiet until the session shrinks below the size it had after the commit, so it cannot fire on every reply.

When a checkpoint fires the REPL prints a line starting with a paperclip emoji:

📎 Checkpointed 7 tool results (~52500 tokens) to the session; the tool loop continues.

The committed bytes are identical to what was already in the request, so the provider's prompt cache is not disturbed by the checkpoint itself.

Note: Checkpoints only run when a session is active and the request is a normal one (not a dry run, a regenerate, or a .continue).

When a request is rejected anyway

If the provider rejects a request with a 400 error whose message says the prompt is too long, Coyote compresses once and retries the same request, provided the session has at least one user message. Phrases recognized include "prompt is too long", "context_length_exceeded", "maximum context length", "input is too long", "too many tokens", "exceeds the context window", "exceeds the maximum number of input tokens", "exceed max_input_tokens" and "request_too_large". Only one retry is made per input.

If compression itself fails, Coyote prints a warning on stderr starting with ⚠️ Compression failed: followed by the reason, and the original error is returned.

Manual controls

Command or setting	Effect
.compress session	Compress now, using the configured compression_keep_last as the upper bound.
.compress session N	Compress now, using N as the upper bound instead. .compress session 0 folds everything.
.set compression_keep_last N	Inside a session: stores a session-level override, saved in the session YAML, that beats the agent and global values. Outside a session: changes the global value.
.set compression_keep_last null	Inside a session: clears the session-level override. Outside a session: resets the global value to 0.
.info and .info session	Show the effective compression settings, the derived limits and, inside a session, the live numbers (see "Reading the numbers" below).
compression_model	Picks a different (usually cheaper) model as the summarizer.

If the kept tail would be the whole session, .compress session refuses with:

Nothing to compress: the kept tail is the whole session; lower compression_keep_last or run `.compress session 0`

Agents can override compression_threshold, compression_keep_last and compression_model in their own config.yaml. compression_safety_valve is global only. When auto-compression fires the REPL prints 📢 Compressing the session.; a manual .compress session prints ✓ Successfully compressed the session. when it finishes.

Reading the numbers

With no session, role or agent active, .info prints a compression_* block of settings and derived limits. Inside a session, .info session prints the same block with four live rows added, next to the session's total_tokens, max_input_tokens and total/max rows, and the YAML that .info exports nests the same values under compression: without the prefix. The first nine rows are settings and the limits derived from them; the live rows describe what a compression right now would do.

Row	What it means	How to use it
compression_threshold	The effective threshold: the session override if set, otherwise the global value.	Compare with total_tokens to see how far the session is from the threshold trigger.
compression_keep_last	The effective upper bound on kept messages: session, then agent, then global.	Confirms which override is in force.
compression_safety_valve	The global valve ratio.	0 means the valve is off.
compression_model	The summarizer model: agent, then global. null means the session model summarizes.	Check this before blaming the session model for a bad summary.
compression_valve_limit	Safety valve times the model window. null when the model declares no window, so the valve cannot fire.	The prompt size at which compression is forced regardless of the threshold.
compression_keep_tokens	The token budget for the kept tail. unlimited when neither the threshold nor the valve defines one.	If the tail keeps getting cut shorter than compression_keep_last, this is why.
compression_checkpoint_tokens	In-flight tool activity at or above this is committed to the session mid-loop.	Long tool loops that never checkpoint have not reached this size.
compression_min_reclaim	The threshold trigger needs at least this many reclaimable tokens. null when the threshold is 0.	Explains why a session over the threshold has not compressed yet.
compression_last_prompt_tokens	The prompt size the provider reported for the most recent request. null until a request has completed.	Compare with total_tokens to see how far the local estimate runs low.
compression_reclaimable	How many tokens a compression right now would fold.	0 means .compress session would refuse.
compression_kept_tail_messages	How many messages would survive a compression right now.	Together with the next row, the shape of the tail you would keep.
compression_kept_tail_tokens	The estimated size of that tail.	Near compression_keep_tokens means the budget, not compression_keep_last, is cutting the tail.
compression_archived_messages	Messages already folded into summaries and archived in the session file.	A rough count of how much history lives only in summaries.

How to read the block:

• total_tokens is well below compression_threshold but compression_last_prompt_tokens is close to compression_valve_limit: the local estimate is running low and the valve will fire first. Lower the threshold so compression runs on your terms.
• compression_reclaimable is small relative to compression_min_reclaim right after a compression: the system prompt plus the kept tail sits close to the threshold. Raise the threshold or lower compression_keep_last.
• compression_kept_tail_tokens is near compression_keep_tokens: the tail is being cut by the budget, not by compression_keep_last. Raising compression_keep_last will not keep more; raising the threshold will.
• compression_reclaimable is 0 with many messages in the session: everything is inside the kept tail. Lower compression_keep_last or run .compress session 0.
• compression_archived_messages keeps climbing while total_tokens barely drops after each compression: the summaries themselves are growing. Try a cheaper, terser compression_model or a shorter summarization_prompt.

Walkthrough

Take a model with a 200,000-token window and this configuration:

compression_threshold: 100000
compression_safety_valve: 0.8
compression_keep_last: 10

The derived numbers are:

• Valve limit: 0.8 x 200,000 = 160,000 tokens.
• Token budget: min(100,000 / 2, 160,000 / 4) = min(50,000, 40,000) = 40,000 tokens.
• Checkpoint size: 200,000 / 4 = 50,000 tokens.
• Minimum reclaim for the threshold trigger: 100,000 / 5 = 20,000 tokens.

Step 1: a plain question

You ask a question and the model answers without calling any tools. The system prompt is 2,000 tokens, your question 300 and the answer 700, so the session is 3,000 tokens. Both triggers are evaluated before the request and again when the turn ends:

• Threshold: 3,000 is not above 100,000. Nothing happens.
• Valve: 3,000 is below 160,000. Nothing happens.
• With compression_keep_last 10, the tail is the whole session anyway, so there is nothing to fold.

Repeat this kind of exchange a couple of dozen times and the session reaches about 58,000 tokens, still under the threshold. Compression has not run.

Step 2: a big job and a tool loop

You now ask the model to do a large job. It starts calling tools, and each round of tool results adds about 7,500 tokens to the in-flight pile. After each reply that contains tool calls, Coyote checks the pile:

• After reply 1: pile 7,500. Above 2,000 but below 50,000. Session 58,000 + pile 7,500 = 65,500, below 160,000. No checkpoint.
• After replies 2 to 6: pile 15,000, 22,500, 30,000, 37,500, 45,000. Same result each time.
• After reply 7: pile 52,500. That is at least 50,000, so the checkpoint fires.

Three messages land in the session: your big-job prompt (about 200 tokens), one tool message holding all seven rounds (about 52,500 tokens) and the checkpoint note. The REPL prints 📎 Checkpointed 7 tool results (~52500 tokens) to the session; the tool loop continues. The session is now about 110,000 tokens. The loop continues with the internal "continue where you left off" prompt.

Step 3: the next request compresses

Before sending the continuation request, Coyote runs the pre-request check and now sees the committed tool message.

• Threshold: 110,000 is above 100,000. Passes.
• Tail selection: the last 10 messages include the checkpoint note, the 52,500-token tool message, the big-job prompt and the seven messages before them. The start moves forward to the first user message in that range. That tail is larger than the 40,000 budget, so the oldest exchange is dropped, and again, until the tail starts at the big-job prompt. That exchange alone is about 52,700 tokens, still over 40,000, so the start moves forward once more. There is no later user message, so the tail is empty.
• Reclaimable: everything after the system prompt, about 108,000 tokens. That is well over the 20,000 minimum, so compression runs and the REPL prints 📢 Compressing the session.

The 108,000 tokens are cut into chunks of at most 60% of the summarizer's window (120,000 tokens for a 200,000-token summarizer, so one chunk here) and summarized.

Step 4: after compression

The session now holds one system message: the original system prompt plus the summary of the whole conversation, including the seven rounds of tool work. The folded messages are archived in the session file.

Because the tail was empty, Coyote considers re-attaching the most recent exchange (the big-job prompt and its tool results) to the next request, but at 52,700 tokens it does not fit the 40,000 budget, so it is not re-attached. The continuation request is the summary system message plus the "continue where you left off" prompt. The model carries on with the job from the summary, and the session is back to a few thousand tokens.

Tuning guide

The trade-off

Compressing rarely keeps more verbatim history, wipes the prompt cache less often, and makes fewer summarizer calls, but each summary call is larger and the per-request cost between compressions is higher. Compressing often makes each request cheaper but loses more detail to summaries, wipes the cache more often, and pays for more summarizer calls.

Rules of thumb

• Set compression_threshold to around 40 to 50 percent of the model window, and not above about 60 percent. The local token estimate can run low (in the case these numbers were calibrated on, by about 1.5x), and the valve needs room to act before the provider rejects the request.
• Leave compression_safety_valve at 0.8.
• Pick a compression_model whose window is comfortably larger than the threshold (each chunk is 60% of the summarizer's window, so at least the threshold divided by 0.6), so a typical fold is a single summarizer request rather than a rolling chain of chunks.
• compression_keep_last of 10 to 20 is plenty. The token budget is usually the real limit on the tail, so a larger number rarely changes what is kept.

Note

These recommendations are starting points. They were calibrated on one observed case where the local estimate was about 1.5x below the provider's real count. Watch how often 📢 Compressing the session. appears in your own sessions and adjust from there.

Starting points

Token budget = min(threshold / 2, 0.8 x window / 4). Checkpoint size = window / 4.

Window	compression_threshold	compression_keep_last	compression_safety_valve	Token budget	Checkpoint size
128,000	60,000	10	0.8	25,600	32,000
200,000	100,000	10	0.8	40,000	50,000
400,000	180,000	15	0.8	80,000	100,000
1,000,000	450,000	20	0.8	200,000	250,000

Symptoms to knobs

• Compression runs every few turns. Raise compression_threshold. If it is already high, the session is probably sitting just above it with little to reclaim; check that your system prompt plus kept tail is not close to the threshold, since the one-fifth reclaim rule is what stops a pointless fold.
• Summaries lose things you need. Raise compression_keep_last so more recent exchanges survive verbatim, or adjust summarization_prompt to tell the summarizer what to preserve.
• A request is rejected as too long right after a compression. The kept tail is too large for the provider's real count. Lower compression_keep_last or compression_safety_valve.
• You want a one-off bigger fold. Run .compress session 0 to fold everything without changing any setting.

See REPL for the full .compress and .set command reference and Sessions for the rest of the session configuration.