Limitations and improvements needed for ai.prompt when bridging OpenAI Chat Completions & Responses APIs

[huleilei]

Hi

First of all, thanks for the amazing work on integrating the ai.prompt function. It's incredibly helpful for batch AI inference. While using it and reviewing the implementation in daft/ai/openai/protocols/prompter.py, I noticed a few limitations regarding how payloads are constructed for the OpenAI APIs (both Chat Completions and the new Responses API).

Because Daft inherently abstracts the content block construction via the _process_message dispatcher, users currently lose access to some critical OpenAI parameters that belong inside the message content block.

Here are the specific issues and missing parameters I observed:

1. Lack of Image detail configuration (Cost & Performance issue)
When passing images (via Numpy or Bytes), Daft hardcodes the block as:

# current implementation
{"type": "image_url", "image_url": {"url": encoded_content}}

Why it matters: OpenAI supports a detail parameter ("low", "high", "auto") inside the image_url / input_image dictionary. For large-scale dataframe processing, being forced into default/high resolution can cause massive, unnecessary token costs. We need a way to pass image_detail="low" (either globally via **options or specifically).

2. Invalid type: "file" format for Chat Completions API
In _build_file_message, when use_chat_completions=True, Daft returns:

{"type": "file", "file": {"filename": ..., "file_data": ...}}

Why it matters: Unlike the Responses API (which accepts input_file), the standard Chat Completions API does not support inline file types in the message content (it only accepts text and image_url). Passing documents this way with use_chat_completions=True will result in a 400 Bad Request from OpenAI.

3. Inability to pass Multi-turn / Few-shot History
Currently, OpenAIPrompter.prompt() forces the structure into exactly one optional system message and exactly one user message:

messages_list.append({"role": "user", "content": content})

Why it matters: For complex batch evaluations or Few-shot learning, we often need to provide historical conversation turns (user -> assistant -> user). The current abstraction blocks users from defining explicit roles for multiple elements in the dataframe expression.

---

Proposed Ideas for Discussion:

• For Image Detail: Could we expose an image_detail parameter in PromptOptions or prompt() signature that gets injected into _build_image_message?
• For Files: Should Daft throw an explicit error/warning when users try to pass general files with use_chat_completions=True, advising them to use the Responses API instead?
• For Multi-turn: Could we support a structured input format (e.g., list of dicts) in the messages expression instead of strictly parsing plain strings/bytes as user content?

I'd love to hear your thoughts on the best way to architect these fixes! I’d be happy to contribute a PR if we can align on an approach.