Controlling hosted image_generation tool parameters at runtime

[CristinaStn]

Summary

We have a gpt-5 agent that has access to the hosted image_generation tool, which is backed by a gpt-image-2 deployment (routed via the x-ms-oai-image-generation-deployment header). We use it through FoundryChatClient.get_image_generation_tool(...). Its parameters (size, quality, output_format, background, ...) are set once, in the tool definition when the agent is built.

We'd like the agent to choose these per request at runtime (e.g. the user asks for a JPEG, or a 1024x1024 high quality image),
but we can't find a supported way to do that with the hosted tool, and would appreciate the recommended pattern.

Concrete example (the core problem)

User query:

Generate a high quality, 1024x1024 size image with output format jpeg about ...

The image generates, but the effective config gpt-image-2 used (read from the image_generation_tool_call content's raw_representation.model_extra) are:

{'background': 'opaque', 'output_format': 'png', 'quality': 'low', 'size': '1254x1254'}

So of four explicit user requests:

• output_format: jpeg → got png (ignored)
• size: 1024x1024 → got 1254x1254 (ignored — and not even one of the documented sizes)
• quality: high → got low (ignored)
• (background not requested) → opaque

Gpt-5 clearly parsed the intent (it's in the user text), but with the hosted tool it has no way to pass output_format/size into the call, and the server-side auto resolution does not honor them. This is our exact pain point.

Question: What is the best practice to use native image generation tool with runtime configurable parameters?

Environment

• agent-framework-core / agent-framework-foundry / agent-framework-openai == 1.10.0
• openai == 2.8.1, azure-ai-projects == 2.2.0
• Client: FoundryChatClient (Azure AI Foundry project endpoint), streaming runs
• Image model: gpt-image-2 deployment, routed via the
  x-ms-oai-image-generation-deployment header on the OpenAI client

[CristinaStn]

Additionally, I would also require a support for editing a specific previously-generated image with the hosted image_generation tool.
Single-turn generation and editing of a just-uploaded image work. What we can't do cleanly is edit a specific image the agent generated on an earlier turn — the model regenerates a new image instead of editing the previous one, and the only workaround we've found (re-attaching images as input_image) does not scale.
thread previous_response_id ourselves)

Concrete scenario

1. User: "generate an image of a dinosaur in an airplane holding a red roses" → the
   hosted tool generates image A.
2. User (next turn): "make the dinosaur red" → the agent generates a brand
   new image of a red dinosaur, rather than editing image A.

Expected: image A, edited (dinosaur recolored red), preserving the original composition.

Question: is there any recommended pattern for such flow?

[moonbox3]

Hi @CristinaStn,

Today the hosted image_generation tool is configurable by the host, but not in the same way a FunctionTool is configurable by the model.

In the Python path, FoundryChatClient.get_image_generation_tool(...) builds a hosted tool configuration with fields like size, quality, output_format, background, etc. That object is then passed through to the Responses API as a hosted tool. Those fields are not exposed to the model as a per-call argument schema, so the model can decide to use the image tool, but it cannot emit size="1024x1024" or output_format="jpeg" as structured tool-call arguments for that hosted tool.

So if you need deterministic runtime control, the recommended patterns are:

1. Host-selected settings: parse/validate the user's requested image settings in your app, then build the hosted image tool for that run with those values.

image_tool = FoundryChatClient.get_image_generation_tool(
    model="gpt-image-2",
    size="1024x1024",
    quality="high",
    output_format="jpeg",
)

result = await agent.run(
    user_message,
    stream=True,
    tools=[image_tool],
)

I would avoid also registering a static default image-generation tool on the agent if you want to vary it per request; create/pass the configured hosted tool for the specific run.

2. Model-selected settings: expose your own FunctionTool with arguments such as prompt, size, quality, output_format, and background, then have that function validate policy/RBAC and call your image-generation backend. That gives the model a real schema to fill in, but the execution is host-owned rather than the native hosted image_generation tool.

For editing a previously generated image, the current practical pattern is similar: persist the generated image artifact in your application and include the specific image again as input when asking for an edit. Agent Framework parses image-generation results as image content, and image Content.from_uri(...) / Content.from_data(...) inputs are serialized back to the Responses API as input_image. There is not currently a higher-level Agent Framework API like "edit image_generation result <id> from a prior turn" that targets a previous generated image by id.

The TL;DR; is: use hosted image_generation when the host can choose the tool configuration up front for the run; use a custom function tool when the model itself needs to choose structured image-generation parameters or when you need stronger control over edit targeting.

[CristinaStn]

Thank you — this is really helpful. Both patterns make sense.
On multi-turn editing: we implemented the recommended application-managed persistence — we keep each generated image (uploaded to Foundry, assistant-… file id) and reattach it as input_image on the edit turn, with the hosted image_generation tool set to action="auto" on a gpt-image-2 deployment. This works most of the time. 👍
However, intermittently the model declines to edit and instead returns a message like:

Sorry—I'm currently unable to apply edits to that photo.
▎ Would you like me to:
▎ - Generate a new image matching the scene, with a yellow helmet and green collar? or
▎ - Give quick step-by-step instructions to edit your photo in a free tool (e.g., Photopea/GIMP)?

The key thing: we see no error on our side — no exception, the run completes normally, the input_image (by Foundry file id) is present in the input, and other turns with the same setup.

Could you help us understand why this happens intermittently?