Skip to content

fix(capabilities): eliminate prefix cache invalidation from in-place system prompt mutation - #282

Merged
Leoyzen merged 3 commits into
mainfrom
fix/prefix-cache-invalidation
Jul 24, 2026
Merged

fix(capabilities): eliminate prefix cache invalidation from in-place system prompt mutation#282
Leoyzen merged 3 commits into
mainfrom
fix/prefix-cache-invalidation

Conversation

@Leoyzen

@Leoyzen Leoyzen commented Jul 24, 2026

Copy link
Copy Markdown
Owner

Problem

MemoryCapability and SkillManagerCap modified SystemPromptPart.content in-place every turn via before_model_request, causing prefix cache misses on every model request. This increased token costs, latency, and wasted compute for both OpenAI (auto-prefix-cache) and Anthropic (prompt caching).

Closes #281.

Root Cause

Two capabilities used a shared _inject_into_system_prompt() utility that mutated SystemPromptPart.content directly in the message history:

# memory.py — _inject_into_system_prompt()
part.content = f"{part.content}\n\n{injected}"

Since the system prompt is part of the message history (not the instructions field), modifying it changes the prefix every turn → cache miss every turn.

Fix

Replace before_model_request in-place mutation with get_instructions returning callables. Pydantic-ai handles callables correctly: literal strings get dynamic=False (cacheable), callable results get dynamic=True (correct for dynamic content). No system prompt mutation occurs.

Changes

File Change
memory.py before_model_requestget_instructions returning async callable. Removed _inject_into_system_prompt + _is_model_request utilities.
skill_manager_cap.py before_model_requestget_instructions returning [static_metadata, dynamic_callable]. Metadata is dynamic=False (cacheable); matched <skill_content> blocks are dynamic=True. Matcher uses RunContext.messages.
combined_toolset.py Fixed get_instructions to preserve callables and sequences from children (previously silently dropped non-str returns).
dynamic_context.py No change — message compaction is intentional behavior, not a cache invalidation bug.

How it works

Before (broken):

  1. get_instructions() returns static metadata string
  2. before_model_request() mutates SystemPromptPart.content in-place with dynamic content
  3. System prompt changes every turn → prefix cache miss

After (fixed):

  1. get_instructions() returns [static_str, async_callable]
  2. Pydantic-ai calls the callable at run time with RunContext (has .messages)
  3. Static string → InstructionPart(dynamic=False) → cacheable
  4. Callable result → InstructionPart(dynamic=True) → correctly marked dynamic
  5. No system prompt mutation → prefix cache preserved

Testing

  • 48 tests pass (6 new tests for dynamic callable, matcher_fn, always_active, callable handling in CombinedToolsetCapability)
  • ruff check clean
  • ruff format --check clean
  • mypy clean

Leoyzen added 2 commits July 24, 2026 17:46
Two fixes for subagent session ordering issue:

1. identifiers.py: Handle backward clock jumps in _create() — when
   now_ms() returns a lower timestamp than _last_timestamp (NTP
   adjustment on macOS), keep the last timestamp and increment the
   counter instead of resetting. Ensures IDs are always monotonically
   increasing even when the wall clock goes backward.

2. session_routes.py: Sort GET /session/{id}/children by time.created
   (ascending) so OpenCode UI left/right navigation shows child
   sessions in creation order.

3. time_utils.py: Fix misleading docstring — now_ms() uses
   time.time_ns() (CLOCK_REALTIME) which is NOT monotonic.
…system prompt mutation

MemoryCapability and SkillManagerCap modified SystemPromptPart.content
in-place every turn via before_model_request, causing prefix cache misses
on every model request (OpenAI auto-cache and Anthropic prompt caching).

- MemoryCapability: replace before_model_request with get_instructions
  returning an async callable (dynamic=True, no system prompt mutation)
- SkillManagerCap: replace before_model_request with get_instructions
  returning [static_metadata, dynamic_callable] — metadata is
  dynamic=False (cacheable), matched skill content is dynamic=True
- CombinedToolsetCapability: fix get_instructions to preserve callables
  and sequences from children (previously silently dropped non-str returns)
- Remove _inject_into_system_prompt utility (no longer used)
- DynamicContextCapability: no change (compaction is intentional)

Closes #281
@gemini-code-assist

Copy link
Copy Markdown

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

Replace before_model_request calls with get_instructions callable
invocation to match the prefix cache fix.
@Leoyzen
Leoyzen merged commit 0e98ed5 into main Jul 24, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Prefix cache invalidation: capabilities modify system prompt and tool returns in-place via before_model_request

1 participant