Tool Use: decouple execution and formalize function calls and responses

[michaelwasserman]

Background

The initial design integrates JS tool execution within the prompt() call itself. The browser process operates as a hidden agent, looping on (model prediction → tool execution → response observation) until a final text response was ready.

While an agentic closed loop prioritizes API simplicity, it detracts from design goals to offer granular control and direct Language Model interactions through this lower-level API.

Motivation

We'd like to realign initial tool use integrations with Prompt API objectives:

• Provide essential Language Model tool types: i.e. Function Declarations (FD), Function Calls (FC), and Function Responses (FR).
  • These can be used by API clients to inspect, reconstruct, and test session history.
• Offer fine-grained client control over tool execution loops used for agentic integrations.
  • This enables clients to define the looping patterns, error handling, limits, etc.
• Align with patterns established by major LLM APIs (OpenAI, Gemini, Claude).
  • This empower clients to use the Prompt API more interchangeably with server-based APIs.

Proposal

We propose a shift from an agentic, closed-loop model to an API-centric, open-loop model for tool execution in the Prompt API. The prompt() method should return a structured Function Call object to the client, which then manages the execution and Function Response feedback loop, maximizing developer control and observability. Function Declarations also need not provide execute functions for now.

The API should initially provide more granular functionally, as ease-of-use can be more readily provided by client libraries and future API enhancements.

Design Principle	Closed-Loop (Original)	Open-Loop (Proposed)
Developer Control	Limited; tool execution is encapsulated; interventions are ad-hoc in tool bodies.	Full; client controls logic, guardrails, and error handling.
Debuggability	Poor; trajectory of tool calls is hidden; clients define bespoke call and response representations.	Excellent; Each tool call and response is a discrete step handled by the client.
Industry Alignment	Divergent; forces custom agent code.	Aligned; Improves portability of agentic code across client-side and server-side models.

Specific Changes

To enable the open-loop model, Function Call and Function Response must be formalized as first-class elements in the API message flow:

Riffing @FrankLi-MSFT 's prototype and @jingyun19's design, declaration would look like:

// The declaration for a tool that a language model can invoke.
dictionary LanguageModelToolDeclaration {
  required DOMString name;
  required DOMString description;
  // JSON schema for the input parameters.
  required object inputSchema;
  // Maybe add an optional/informational responseSchema per #137?
};

dictionary LanguageModelCreateCoreOptions {
  ....
  // Tools that the language model can use.
  sequence<LanguageModelTool> tools;
};

Then, tool calls and responses are made accessible to the API client. This definitely needs some workshopping!

// Represents a tool call requested by the language model.
dictionary LanguageModelToolCall {
  // Unique identifier for this tool call within the session.
  required DOMString callID;
  required DOMString name;
  // An object fitting the JSON input schema for the tool.
  object input;
};

// Represents the response from executing a tool call.
dictionary LanguageModelToolResponse {
  // Matches the callID from the corresponding tool call.
  required DOMString callID;
  // Response from tool execution. (using an object, rather than a string, per #138)
  // Maybe this should be a sequence of LanguageModelMessageContent?
  required object response;
  // Maybe add explicit success/error signals?
};

enum LanguageModelMessageType { "text", "image", "audio", "tool-call", "tool-response" };

typedef (
  ImageBitmapSource
  or AudioBuffer
  or HTMLAudioElement
  or BufferSource
  or DOMString
  or LanguageModelToolCall  // NEW
  or LanguageModelToolResponse  // NEW
) LanguageModelMessageValue;

// NEW: When `expectedOutputs` has non-text (e.g. `tool-call`), this now yields a content dictionary sequence.
Promise<DOMString or sequence<LanguageModelMessageContent>> prompt(
    LanguageModelPrompt input,
    optional LanguageModelPromptOptions options = {}
  );

// NEW: When `expectedOutputs` has non-text (e.g. `tool-call`), stream chunks are now content dictionaries.
ReadableStream promptStreaming(
  LanguageModelPrompt input,
  optional LanguageModelPromptOptions options = {}
);

We'll need to resolve a number of finer points, e.g.

• Behavior when supplying responseConstraint (maybe those requests won't yield tool calls?).
• Behavior when tools are declared but expectedOutputs does not contain tool-call (same or don't yield tool calls?).
• Whether LanguageModelMessageValue should be split to represent model inputs and outputs.
• Behavior when responses are not input immediately after tool calls (do warnings suffice, is prompt queueing a footgun?).
• Ability to specify tool call frequency or specific occurrences amid constrained responses.
• Probably a lot more!

In any case, the design aims provide the necessary low-level building blocks for developers to construct robust, observable, and fully customized tool use workflows. This proposal is still flexible and we're seeking design feedback!

Much credit for explorations should be given to:

• @FrankLi-MSFT (leading initial implementation design and prototyping)
• @jingyun19 (Exploring design and scope revisions, welcome to Chrome!)

[michaelwasserman]

FYI @anssiko and @reillyeon

[reillyeon]

Internally we've also discussed an alternate design for LanguageModelToolCall which makes it an interface with a setResponse() method:

// Represents a tool call requested by the language model.
interface LanguageModelToolCall {
  readonly attribute DOMString name;
  // An object fitting the JSON input schema for the tool.
  readonly attribute object input;

  void setResponse(object response);
};

In this design the tool call responses are implicitly consumed by the next call to prompt() or promptStreaming() instead of being passed explicitly by the developer. This removes the need for a callID property and gives implementations some flexibility to prompt the model with tool responses in the order the model expects. It does however add back some of the complexity to the API that the design above attempts to avoid by removing the tool callbacks.

[nathanmemmott]

Here's some thoughts I have on this. I ended up exploring what the closed loop solution looked like. I want to explore the open loop solution more later.

Requirements

To break down key functionalities that we're looking for:

• We want to be able to have multiple tool calls. Possibly multiple of the same tool call
• We want the developer to be able to return the results in a way that communicates which result belongs to which of these tool calls.
• We want the developer to know when all tool calls of a set of tool calls has been called
• We want to be open to the possibility that a developer doesn't respond with results. This could mean several things:
  • The developer communicates:
    • The tool call was cancelled or errored
    • The tool call will be answered later after the model has received other input and given other output
  • The developer doesn't communicate:
    • Ignores the tool call and never gives a response
    • Ignores the tool call but gives a response later

Another thing to keep in mind when designing this is that tool calling is a form of multimodal output, so we should try to consider how we'd like to align with other future multimodal outputs, e.g. audio and image.

And is our goal to be ergnomic? Or to provide full control of the API? I believe our goal should be to make it as ergonomic as possible without compromising on giving control to developers.

Basic structure

I think that a good starting point is to have both prompt and promptStreaming return/stream chunks of this shape:

type chunk = {
   type: "text" | "toolcall" | "image" | "audio"
   ...other properties that could either be shared or different between the types
}

Correlating responses with their tool calls

This is a big question I see in a lot of our discussion: how does the developer say that this response is for this tool call?

Here are some ways that have been proposed:

• As in the original proposal, execute the tool call itself and its result is the response
• Have the developer return an array of results which would have to be in the same order as the order of the tool calls
  • If the developer wants to answer the tool call later, how would we identify it then?
• Have the developer return an object with the tool call names
  • This would mean the model could never call the same tool call twice in one batch since there would be a naming collision
  • If the developer wants to answer the tool call later, how would we identify it then?
• Give the developer a unique id and have them pass it back their results in an object where the keys are the ids
• Give the developer an object that represents the set of tool calls and they can set the response on it

What's wrong with the original proposal?

Could we meet all these requirements and stick to the original design of the API automatically calling the provided executor functions? The original design provides a nice out of the box experience for developers who just want to plug and play. And if power users are able to get what they want out of it, then it could be the best of both worlds.

The main problem that I see about the current design is that there's no signal that there's not another tool call coming. But we could easily provide that signal by providing a chunk with type "end_of_toolcall". Given that signal, a power user could easily collect their toolcalls via a helper:

class ToolCallCollector {
  toolCalls = [];

  createTool({ name, description, jsonSchema }) {
    return {
      name, description, jsonSchema,
      execute: (...args) => {
        const { promise, resolve } = Promise.withResolvers();
        this.toolCalls.push({ name, args, resolve });
        return promise;
      }
    }
  }
}

const collector = new ToolCallCollector();
const session = await LanguageModel.create({
  tools: [
    collector.createTool({name: 'getWeather', ...}),
    collector.createTool({name: 'getTime', ...}),
  ]
});

const stream =
  session.promptStreaming('What\'s the current weather and time?');

for await (const chunk of stream) {
  switch (chunk.type) {
    case "end_of_toolcall":
      handleToolCalls(collector.toolCalls);
      break;
    case "text":
      console.log(chunk.value);
      break;
  }  
}

This however forces the developer to communicate a decision about how they want to handle a tool call. Which could be a good thing. They would have to decide to either:

• Return a value which immediately gives the model a result
• Return a Promise which forces the browser to decide whether to await it or continue streaming and get the result later
  • Browser implementation choice
• Throw an error, which would be a way to cancel the result

For more explicit control, we could provide some primitives they could return like IgnoreResponse or AsyncResponse.

[FrankLi-MSFT]

fyi: @sushraja-msft