Allow customization of tool validation error messages

[sigglep]

Problem

When tool calls fail validation, Pydantic AI sends "2 validation errors: ... Fix the errors and try again" to the LLM. Some models interpret this as user input rather than an internal error, causing confused responses.

Proposed

Add a way to customize/override the validation error message format. For example with a callable.
Even better if we'd be able to inject RunContext into that formatter so we could do conditionals based on what the state of the conversation is.

class Agent:
    def __init__(
        self,
        ...,`
        tool_retry_formatter: Callable[[str, RunContext], str] | None = None`
    ):
        ...

# Usage
def format_tool_error(error_message: str, ctx: RunContext) -> str:
    return f"Customizable text that tells the agent exactly what to do, that couldn't be explained pre-validation failure\n{error_message}"

Use Case

Prepending custom error

Benefits

• Clearer error context for LLMs (being able to inform that THIS IS NOT THE USER)
• Custom agent behavior on validation failures

References

Slack thread

[DouweM]

@sigglep Really, every hard-coded string in Pydantic AI that is sent to the model should be overridable, possibly contextually as you say. So I want to come up with a generic approach to specify formatters like this, instead of giving each a dedicated arg on Agent().

The only hard-coded string that is currently overridable is the prompt used by PromptedOutput, which can be specified per model via ModelProfile:

pydantic-ai/pydantic_ai_slim/pydantic_ai/profiles/__init__.py

Lines 43 to 52 in 6bdd39b

	prompted_output_template: str = dedent(
	"""
	Always respond with a JSON object that's compatible with this schema:
	{schema}
	Don't include any text or Markdown fencing before or after.
	"""
	)
	"""The instructions template to use for prompted structured output. The '{schema}' placeholder will be replaced with the JSON schema for the output."""

That currently takes a string with a {schema} placeholder, but could also have been implemented as a Callable[[RunContext, JsonSchema], str], or we could support both.

The retry message logic lives here:

pydantic-ai/pydantic_ai_slim/pydantic_ai/messages.py

Lines 935 to 948 in 6bdd39b

	def model_response(self) -> str:
	"""Return a string message describing why the retry is requested."""
	if isinstance(self.content, str):
	if self.tool_name is None:
	description = f'Validation feedback:\n{self.content}'
	else:
	description = self.content
	else:
	json_errors = error_details_ta.dump_json(self.content, exclude={'__all__': {'ctx'}}, indent=2)
	plural = isinstance(self.content, list) and len(self.content) != 1
	description = (
	f'{len(self.content)} validation error{"s" if plural else ""}:\n```json\n{json_errors.decode()}\n```'
	)
	return f'{description}\n\nFix the errors and try again.'

It's used for both failed tool calls and ModelRetry errors raised by e.g. output functions (which aren't necessarily handled using output tool calls). It'd make sense to have separate templates/callables for each.

I can't immediately think of other hard-coded strings we have, but there could easily be more in the future, e.g. for agent handoff and "code mode".

So I think we should have a new dataclass that has fields for all of these templates, and then give Agent (and agent.run and agent.override?) a single argument to pass in that object. What do you think it should be called? prompt=Prompts(...) is not clear enough. prompt_templates=PromptTemplates(...) is slightly more suggestive of this being for reusable templates, but maybe you have a better idea.

[sigglep]

I think prompt_templates is fine!
It put's some distance between itself and the "message_history" and "history_processors".

It's kind of like an "inject-this-instead-of-what-we've-hardcoded-to-inject"-parameter.
Prompt_templates points at the fact that we're creating templates that transform the prompt right before the agent receives it.

I do feel like it would be a missed opportunity to not enable dynamically set prompt_templates based on whatever-the-user-decides though.

If dynamically set PromptTemplates was enabled I guess we'd need to target ModelRetry e.g with a callable.

I guess a question then becomes, how can one link the formatter hard to the target of the formatter (ModelRetry etc.), so that the user (me) doesn't confuse the model_retry_'formatter' with something else? I guess the signature of the data-class could define the relationship.

Like:

@dataclass
class PromptTemplates:
    """Templates for specific message parts that Pydantic AI injects.
    
    Each template can be either:
    - A static string: Simple replacement for the default message
    - A callable: Dynamic formatting based on RunContext
    """

    model_retry: str | Callable[[ModelRetry, RunContext], str] | None = None
    """Template for ModelRetry messages when model output validation fails."""

    tool_return: str | Callable[[ToolReturnPart, RunContext], str] | None = None
    """Template for ToolReturnPart messages when tools return results or errors."""

    retry_prompt: str | Callable[[RetryPromptPart, RunContext], str] | None = None
    """Template for RetryPromptPart messages when injecting retry instructions."""

And usage would be:

 agent = Agent(
      model='openai:gpt-5',
      prompt_templates=PromptTemplates(
          tool_return="Nice work Mr. Agent here's the data you requested: ",
          model_retry=lambda msg, ctx: (
              f"Have you read the documentation with the read_document tool?\n\n{msg.content}"
              if 'read_document' not in {call.tool_name for call in ctx.tool_calls}
              else msg.content
          ),
      )
  )

Another pro of having callables would be the fact that the developer can trust pydantic-ai, but still do touches to the injected prompt. Like:

  agent = Agent(
      model='openai:gpt-4',
      prompt_templates=PromptTemplates(
          model_retry=lambda msg, ctx: f"IMPORTANT: {msg.content}",  # Keep default, add emphasis
      )
  )

[DouweM]

@sigglep Yeah I like PromptTemplates with RunContext-aware callables. We can start by having it on Agent() but I think it'd make sense on agent.run and agent.override as well.

The way that this would need to be implemented is by "compiling" the text for each RetryPromptPart in the message history around here, where RunContext is still available:

pydantic-ai/pydantic_ai_slim/pydantic_ai/_agent_graph.py

Lines 490 to 500 in c27e5e4

	original_history = ctx.state.message_history[:]
	message_history = await _process_message_history(original_history, ctx.deps.history_processors, run_context)
	# `ctx.state.message_history` is the same list used by `capture_run_messages`, so we should replace its contents, not the reference
	ctx.state.message_history[:] = message_history
	# Update the new message index to ensure `result.new_messages()` returns the correct messages
	ctx.deps.new_message_index -= len(original_history) - len(message_history)
	# Merge possible consecutive trailing `ModelRequest`s into one, with tool call parts before user parts,
	# but don't store it in the message history on state. This is just for the benefit of model classes that want clear user/assistant boundaries.
	# See `tests/test_tools.py::test_parallel_tool_return_with_deferred` for an example where this is necessary
	message_history = _clean_message_history(message_history)

And then storing that text on the part (without modifying the original object, so we'd need to copy it) before sending the message history into Model.request. Then when the model calls RetryPromptPart.model_response() it should see if there's already a "compiled" text stored on it, and prefer that over generating it anew.

Note that for #3179, we also need to support strings that an optimizer can iterate on, so considering the branching that happens in RetryPromptPart.model_response(), we'll likely need a separate key for each of those scenarios.

Would you be up for trying to contribute this new feature?