Claude can now accept system prompt updates part-way through a conversation

[simonw]

https://platform.claude.com/docs/en/about-claude/models/whats-new-claude-4-8

Mid-conversation system messages

Claude Opus 4.8 accepts role: "system" messages immediately after a user turn in the messages array (subject to placement rules). This lets you append updated instructions later in a long-running conversation without restating the full system prompt, which preserves prompt cache hits on the earlier turns and reduces input cost on agentic loops. No beta header is required. See Mid-conversation system messages for usage details.

See also the Anthropic SDK here: anthropics/anthropic-sdk-python@2b82676#diff-a947c9c02eab58e8ddbe799a11832d533836d242e07c7251997f8543f0981f2f

The good news is we're most of the way there with the new messages= and parts stuff already:

https://github.com/simonw/llm/blob/be27b91aa83142ee080c467976dc47fa4b5fb7ec/llm/parts.py#L274-L280

def system(*items: Any, provider_metadata: Optional[Dict[str, Any]] = None) -> Message:
    "Build a Message with role='system'."
    return Message(
        role="system",
        parts=normalize_parts(items),
        provider_metadata=provider_metadata,
    )

So it may be that we don't have to do anything special beyond shipping that as a non-alpha and supporting in llm-anthropic.

[simonw]

Huh, those placement rule limitations are interesting:

Limitations

• Not for the first message. A system message cannot be the first entry in messages. Use the top-level system field for instructions that apply from the very start.
• Placement is constrained. A system message must immediately follow a user turn (or an assistant turn ending in server tool use) and must precede an assistant turn or end the array. Placing it elsewhere returns a 400 error.