Skip to content

Enforce the hard context limit on the full request, with turn rollback - #23

Merged
lavindeep merged 5 commits into
mainfrom
fix/context-budget-estimator
Jul 10, 2026
Merged

Enforce the hard context limit on the full request, with turn rollback#23
lavindeep merged 5 commits into
mainfrom
fix/context-budget-estimator

Conversation

@lavindeep

@lavindeep lavindeep commented Jul 10, 2026

Copy link
Copy Markdown
Owner

Problem

The context-budget estimate that drives compaction and the hard-limit refusal
only counted the system prompt and history. It ignored the encoded tool
schemas and tool-call arguments — roughly a couple thousand tokens on a
default toolset — so a request that looked under the limit on paper could
still overrun once everything was serialized to the model. There was also no
in-loop guard: a turn that only crossed the limit after several tool results
accumulated was sent anyway, and a refused turn could leave a stuck user
message (or an orphaned assistant tool_call) in history and on --resume.

Change

  • Full-request estimate. estimated_prompt_tokens() now includes the live
    profile's encoded tool schemas (tool_schema_tokens) and per-message
    tool-call arguments, closing the undercount. run_command results are held
    to the command prompt cap.
  • Gate before every model call. The pre-turn gate and a new in-loop gate
    both check the full request against hard_limit_tokens. With automatic
    compaction on, the runtime compacts first and refuses only when compaction
    cannot recover; with it off, it refuses directly.
  • Turn rollback. A refused pre-turn request discards the just-recorded user
    message (discard_last_message reconciliation record) so --resume does not
    restore a stuck user turn with no reply. A mid-loop overrun rolls the
    in-flight assistant turn back (last assistant message to end of history,
    truncate_last_turn) so no orphaned tool_call is re-sent.
  • Last-resort recovery (auto_compact only). When an in-flight tool result is
    large enough to overrun on its own inside the tool loop, the runtime
    force-digests every tool result — including the normally protected recent
    window — and retries once before refusing. This is a bounded, documented
    exception to the protected-tail compaction invariant, gated behind
    auto_compact; with compaction off the protected tail is never rewritten.

Behavior change

Turns that would exceed the hard context limit are now refused deterministically
before the model is called (previously such a request could still be sent).
On refusal the user sees a guidance status message, the offending turn is rolled
back out of history and the session transcript, and the session stays usable
rather than being pinned over budget. This applies whether automatic compaction
is on or off.

Docs

docs/DESIGN.md updated in step: the budget section now describes tool-schema
counting and compaction-on/off refusal; §20.2 documents the bounded
force-digest recovery exception; the Session-resume row documents the new
discard_last_message reconciliation record.

Testing

Full local phase gate green at HEAD:

  • ruff check . — passed
  • ruff format --check . — clean (146 files)
  • mypy shellpilot --strict — Success, 83 source files
  • pytest — 1632 passed

New coverage includes: schemas + tool-call args in the estimate; explicit
context setting not capped; refusal when compaction cannot recover; refusal
discarding the recorded user message; mid-loop rollback of the in-flight turn;
mid-loop refusal with auto_compact off not digesting the protected tail;
discard_last_message load semantics; and run_command honoring the command
prompt cap.

Include tool schemas and tool-call arguments in the live estimate, compact at the soft threshold, refuse when the hard limit still cannot be met, and apply the command prompt cap to run_command results.
Repository owner deleted a comment from cursor Bot Jul 10, 2026
Refuse oversized turns before they stick in history/session, force-digest
in-flight tool results, and discard incomplete assistant+tool exchanges so
the next turn is not permanently stuck over budget.
Force-digesting protected tool results is compaction; with auto_compact off,
refuse and roll back the in-flight turn without rewriting earlier tool text.
Repository owner deleted a comment from cursor Bot Jul 10, 2026
Repository owner deleted a comment from cursor Bot Jul 10, 2026
@lavindeep
lavindeep force-pushed the fix/context-budget-estimator branch from 44a9201 to 04d65e8 Compare July 10, 2026 11:54
Repository owner deleted a comment from cursor Bot Jul 10, 2026
The hard-limit stop returns an empty assistant Message directly, so the
last_reply capture in the tool loop was never read (ruff F841). Remove it
and apply the formatter to the hard-limit condition and the recovered check.
Document that the pre-turn and in-loop gates now count tool schemas and
refuse (with compaction on or off), the bounded force-digest exception to
the protected-tail compaction invariant, and the discard_last_message
reconciliation record.
@lavindeep
lavindeep force-pushed the fix/context-budget-estimator branch from 04d65e8 to d37d4e8 Compare July 10, 2026 12:23
@lavindeep lavindeep changed the title Count full request tokens before every model call Enforce the hard context limit on the full request, with turn rollback Jul 10, 2026
@lavindeep
lavindeep merged commit a465e3a into main Jul 10, 2026
2 checks passed
@lavindeep
lavindeep deleted the fix/context-budget-estimator branch July 10, 2026 12:31
Repository owner deleted a comment from cursor Bot Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant