@tanstack/openai-base@0.16.0

Minor Changes

• #1837 824ee98 - Add computer use that runs the same code on OpenAI and on Anthropic. A driver controls the machine. The Playwright and sandbox drivers give each thread its own machine.

  New packages:

  • @tanstack/ai-computer-use adds createComputerUse(). It returns tools, a middleware, a safety-check interrupt, and close. Each threadId gets its own machine, action queue, and approvals. A thread that stays idle for idleTimeout (10 minutes by default) closes.
  • @tanstack/ai-computer-playwright adds playwrightComputer(), which opens one browser page for each thread. createPlaywrightComputerDriver() controls one page that you give it. Both add a navigate tool that opens http: and https: URLs.
  • @tanstack/ai-computer-sandbox adds sandboxComputer(), which creates one Linux desktop sandbox for each thread from any @tanstack/ai-sandbox provider, like e2bSandbox({ template: 'desktop' }). createSandboxComputerDriver() controls one sandbox that you give it.
  • @tanstack/ai-computer-local adds localComputer(), which lets the model use the mouse, keyboard, and main screen of the machine that runs chat(). It works on Windows, macOS, and Linux with @jitsi/robotjs.

  Changes in existing packages:

  • @tanstack/ai adds threadId to ToolExecutionContext. It also adds the computer tool metadata reader in @tanstack/ai/adapter-internals.
  • @tanstack/ai-anthropic and @tanstack/openai-base send the generic computer tool to the model as the native computer tool of the provider.
  • @tanstack/ai-openai marks the models that support the computer tool.

  const computerUse = createComputerUse({
    driver: playwrightComputer({ startUrl: 'https://tanstack.com' }),
  })
  
  chat({
    adapter,
    messages,
    threadId,
    tools: computerUse.tools,
    middleware: [computerUse.middleware],
    interrupts: [computerUse.interrupt],
  })
  
  await computerUse.close(threadId) // one thread
  await computerUse.close() // every thread, and the browser

  The ai-core agent skill routes computer use to @tanstack/ai-computer-use and its three drivers.

• #1837 824ee98 - computerUseTool({ type: 'computer' }) adds the computer tool of the newer OpenAI models, for example gpt-6.1-sol. The tool name is computer. Each call has the batched actions of the model and the pending_safety_checks, and the replay sends the actions back as they came. computerUseTool({ type: 'computer_use_preview', ... }) for the computer-use-preview model works as before. The Responses tool choice type also accepts computer. Computer screenshots now go out with detail: 'original', as in OpenAI's computer use guide.

Patch Changes

• #1837 824ee98 - The computer use loop of computerUseTool() runs. A computer_call from the model is now a tool call to computer_use_preview, with the action and the pending_safety_checks as its input. The tool result goes back as computer_call_output, with the first image of the result as the screenshot. When the call does not fail, the output acknowledges the safety checks of the call.

  A failed computer call sends the newest screenshot of the conversation, because the failed call has no screenshot of its own. A computer call output has no text, so the reason for the failure goes to the model in a user message after the call outputs.

  A computer tool result with no image fails the run with an error. Before, the adapter sent an empty screenshot.

• Updated dependencies [824ee98]:

  • @tanstack/ai@0.71.0