Prefix cache invalidation: capabilities modify system prompt and tool returns in-place via before_model_request

[Leoyzen]

Problem

Several capabilities on main modify the LLM request prefix in-place every turn via before_model_request, causing prefix cache misses on every model request. Since the system prompt and tool return content change each turn, providers (OpenAI auto-prefix-cache, Anthropic prompt caching) cannot reuse cached prefixes, leading to:

• Increased token costs (re-processing the full prefix every turn)
• Higher latency (no cache hit on long system prompts)
• Wasted compute on re-encoding identical content

Root Causes

1. SkillManagerCap.before_model_request — modifies SystemPromptPart.content in-place

File: src/agentpool/capabilities/skill_manager_cap.py (L339-402)

Every turn, before_model_request calls _inject_into_system_prompt(messages, injected) which appends <skill_content> blocks to the first SystemPromptPart.content found in messages:

# memory.py:90-108
def _inject_into_system_prompt(messages, injected):
    for msg in messages:
        for part in msg.parts:
            if part.part_kind == "system-prompt":
                if injected not in part.content:
                    part.content = f"{part.content}\n\n{injected}"  # ← in-place modification
                return True

Even with the dedup guard (if injected not in part.content), the first-turn injection changes the system prompt content, and any subsequent change to matched skills (or pruning by other capabilities) re-triggers injection → prefix changes → cache miss.

2. MemoryCapability.before_model_request — same in-place modification

File: src/agentpool/capabilities/memory.py (L67-79)

Identical pattern: calls _inject_into_system_prompt(messages, injected) with memory content. If memories change across turns (extraction adds new entries), the system prompt content changes every turn.

3. DynamicContextCapability.before_model_request — replaces entire message list

File: src/agentpool/capabilities/dynamic_context.py (L59-78)

When len(messages) > threshold_count, replaces request_context.messages with compacted messages. This is expected behavior at high watermark, but it does invalidate the cache. Not a bug per se — documenting for awareness.

4. CombinedToolset.get_instructions() — discards InstructionPart.dynamic flag

File: src/agentpool/capabilities/combined_toolset.py (L165-179)

Collects instructions from children as plain strings, losing any InstructionPart metadata:

def get_instructions(self) -> str | None:
    parts: list[str] = []
    for cap in self._capabilities:
        instr = cap.get_instructions()
        if isinstance(instr, str):
            parts.append(instr)
    return "\n\n".join(parts) if parts else None

pydantic-ai's normalize_toolset_instructions() (_instructions.py:73) marks all plain strings from toolset get_instructions() as dynamic=True:

part = item if isinstance(item, InstructionPart) else InstructionPart(content=item, dynamic=True)

This means all capability instructions are classified as dynamic → Anthropic cannot cache them (cache breakpoint goes before dynamic instructions).

5. All capability get_instructions() return str | None instead of InstructionPart

Files: skill_manager_cap.py, memory.py, subagent_capability.py, mcp_server_cap.py, function_toolset.py, code_mode_capability.py

All return plain strings, which get marked dynamic=True by the framework. Even when content is perfectly stable across turns, Anthropic's explicit prompt caching cannot cache these instructions.

Background: pydantic-ai's Caching Architecture

pydantic-ai has no framework-level caching — get_instructions() and get_tools() are re-evaluated every turn. The InstructionPart.dynamic field is a metadata hint for provider-level caching:

dynamic	Meaning	Anthropic behavior
False	Static/literal instruction	✅ Cache breakpoint after last static part
True	Dynamic/function-sourced	❌ No cache breakpoint

For OpenAI, content stability is what matters — if the instruction string is byte-identical every turn, the auto prefix cache hits regardless of dynamic. For Anthropic, dynamic=False is required for explicit cache breakpoints.

Proposed Fixes

Fix 1: Move skill injection from before_model_request to get_instructions() (P0)

SkillManagerCap should return skill content via get_instructions() as InstructionPart(dynamic=False):

def get_instructions(self) -> InstructionPart | None:
    # ... existing available-skills metadata ...
    # ... add matched skill_content blocks ...
    return InstructionPart(content=full, dynamic=False)

async def before_model_request(self, ctx, request_context):
    return request_context  # no-op

Obstacle: get_instructions() on the toolset path receives a run_context (has message history for matcher), but on the agent construction path it doesn't. Need to handle both cases — e.g., compute injection lazily on first toolset-path call and cache for subsequent turns.

Fix 2: Same pattern for MemoryCapability (P1)

Move memory injection to get_instructions():

• Stable memories → InstructionPart(dynamic=False)
• Changing memories → InstructionPart(dynamic=True) (still better than modifying SystemPromptPart.content in-place)

Fix 3: CombinedToolset.get_instructions() — preserve InstructionPart (P1)

def get_instructions(self) -> InstructionPart | None:
    parts: list[InstructionPart] = []
    for cap in self._capabilities:
        instr = cap.get_instructions()
        if instr is None:
            continue
        if isinstance(instr, InstructionPart):
            parts.append(instr)
        elif isinstance(instr, str):
            parts.append(InstructionPart(content=instr, dynamic=True))
    if not parts:
        return None
    sorted_parts = InstructionPart.sorted(parts)
    return InstructionPart(
        content="\n\n".join(p.content for p in sorted_parts),
        dynamic=any(p.dynamic for p in sorted_parts),
    )

Fix 4: All stable capabilities return InstructionPart(dynamic=False) (P2)

For capabilities whose get_instructions() content is stable within a run:

• subagent_capability.py
• mcp_server_cap.py
• function_toolset.py
• code_mode_capability.py

Fix 5: Deprecate _inject_into_system_prompt() (P2)

After Fix 1 and Fix 2, no callers remain. Remove the utility function from memory.py.

Impact

Capability	Current behavior	After fix
SkillManagerCap	Modifies system prompt every turn	Returns static InstructionPart
MemoryCapability	Modifies system prompt every turn	Returns InstructionPart (static or dynamic)
DynamicContextCapability	Replaces messages at high watermark	No change (expected behavior)
CombinedToolset	Discards dynamic flag	Preserves dynamic flag
All other caps	Plain string → dynamic=True	InstructionPart(dynamic=False)

Environment

• pydantic-ai: 2.9.0
• agentpool: main branch (8215a8b)
• Observed with: xeno-nmt-harness translation team (8 agents, 3 skills, DCP capability)