usecomputer

usecomputer is a desktop automation CLI for AI agents. It works on macOS and Linux (X11).

Screenshot, mouse control (move, click, drag, scroll), and keyboard synthesis (type and press) are all available as CLI commands backed by a native Zig binary — no Node.js runtime required.

Install

npm install -g usecomputer

Requirements

• macOS — Accessibility permission enabled for your terminal app
• Linux — X11 session with DISPLAY set (Wayland via XWayland works too)

Quick start

usecomputer mouse position --json
usecomputer mouse move -x 500 -y 500
usecomputer click -x 500 -y 500 --button left --count 1
usecomputer type "hello"
usecomputer press "cmd+s"

Library usage

import * as usecomputer from 'usecomputer'

const screenshot = await usecomputer.screenshot({
  path: './tmp/shot.png',
  display: null,
  window: null,
  region: null,
  annotate: null,
})

const coordMap = usecomputer.parseCoordMapOrThrow(screenshot.coordMap)
const point = usecomputer.mapPointFromCoordMap({
  point: { x: 400, y: 220 },
  coordMap,
})

await usecomputer.click({
  point,
  button: 'left',
  count: 1,
})

These exported functions intentionally mirror the native command shapes used by the Zig N-API module. Optional native fields are passed as null when absent.

OpenAI computer tool example

import fs from 'node:fs'
import * as usecomputer from 'usecomputer'

async function sendComputerScreenshot() {
  const screenshot = await usecomputer.screenshot({
    path: './tmp/computer-tool.png',
    display: null,
    window: null,
    region: null,
    annotate: null,
  })

  return {
    screenshot,
    imageBase64: await fs.promises.readFile(screenshot.path, 'base64'),
  }
}

async function runComputerAction(action, coordMap) {
  if (action.type === 'click') {
    await usecomputer.click({
      point: usecomputer.mapPointFromCoordMap({
        point: { x: action.x, y: action.y },
        coordMap: usecomputer.parseCoordMapOrThrow(coordMap),
      }),
      button: action.button ?? 'left',
      count: 1,
    })
    return
  }

  if (action.type === 'double_click') {
    await usecomputer.click({
      point: usecomputer.mapPointFromCoordMap({
        point: { x: action.x, y: action.y },
        coordMap: usecomputer.parseCoordMapOrThrow(coordMap),
      }),
      button: action.button ?? 'left',
      count: 2,
    })
    return
  }

  if (action.type === 'scroll') {
    await usecomputer.scroll({
      direction: action.scrollY && action.scrollY < 0 ? 'up' : 'down',
      amount: Math.abs(action.scrollY ?? 0),
      at: typeof action.x === 'number' && typeof action.y === 'number'
        ? usecomputer.mapPointFromCoordMap({
            point: { x: action.x, y: action.y },
            coordMap: usecomputer.parseCoordMapOrThrow(coordMap),
          })
        : null,
    })
    return
  }

  if (action.type === 'keypress') {
    await usecomputer.press({
      key: action.keys.join('+'),
      count: 1,
      delayMs: null,
    })
    return
  }

  if (action.type === 'type') {
    await usecomputer.typeText({
      text: action.text,
      delayMs: null,
    })
}
}

Anthropic computer use example

Anthropic's computer tool uses action names like left_click, double_click, mouse_move, key, type, scroll, and screenshot. usecomputer provides the execution layer for those actions.

import fs from 'node:fs'
import Anthropic from '@anthropic-ai/sdk'
import type {
  BetaToolResultBlockParam,
  BetaToolUseBlock,
} from '@anthropic-ai/sdk/resources/beta/messages/messages'
import * as usecomputer from 'usecomputer'

const anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY })

const message = await anthropic.beta.messages.create({
  model: 'claude-opus-4-6',
  max_tokens: 1024,
  tools: [
    {
      type: 'computer_20251124',
      name: 'computer',
      display_width_px: 1024,
      display_height_px: 768,
      display_number: 1,
    },
  ],
  messages: [{ role: 'user', content: 'Open Safari and search for usecomputer.' }],
  betas: ['computer-use-2025-11-24'],
})

for (const block of message.content) {
  if (block.type !== 'tool_use' || block.name !== 'computer') {
    continue
  }

  const toolUse = block as BetaToolUseBlock
  await usecomputer.screenshot({
    path: './tmp/claude-current-screen.png',
    display: null,
    window: null,
    region: null,
    annotate: null,
  })
  const coordinate = Array.isArray(toolUse.input.coordinate)
    ? toolUse.input.coordinate
    : null
  const point = coordinate
    ? { x: coordinate[0] ?? 0, y: coordinate[1] ?? 0 }
    : null

  switch (toolUse.input.action) {
    case 'screenshot': {
      break
    }
    case 'left_click': {
      if (point) {
        await usecomputer.click({ point, button: 'left', count: 1 })
      }
      break
    }
    case 'double_click': {
      if (point) {
        await usecomputer.click({ point, button: 'left', count: 2 })
      }
      break
    }
    case 'mouse_move': {
      if (point) {
        await usecomputer.mouseMove(point)
      }
      break
    }
    case 'type': {
      if (typeof toolUse.input.text === 'string') {
        await usecomputer.typeText({ text: toolUse.input.text, delayMs: null })
      }
      break
    }
    case 'key': {
      if (typeof toolUse.input.text === 'string') {
        await usecomputer.press({ key: toolUse.input.text, count: 1, delayMs: null })
      }
      break
    }
    case 'scroll': {
      await usecomputer.scroll({
        direction: toolUse.input.scroll_direction === 'up' || toolUse.input.scroll_direction === 'down' || toolUse.input.scroll_direction === 'left' || toolUse.input.scroll_direction === 'right'
          ? toolUse.input.scroll_direction
          : 'down',
        amount: typeof toolUse.input.scroll_amount === 'number' ? toolUse.input.scroll_amount : 3,
        at: point,
      })
      break
    }
    default: {
      throw new Error(`Unsupported Claude computer action: ${String(toolUse.input.action)}`)
    }
  }

  const afterActionScreenshot = await usecomputer.screenshot({
    path: './tmp/claude-computer-tool.png',
    display: null,
    window: null,
    region: null,
    annotate: null,
  })
  const imageBase64 = await fs.promises.readFile(afterActionScreenshot.path, 'base64')
  const toolResult: BetaToolResultBlockParam = {
    type: 'tool_result',
    tool_use_id: toolUse.id,
    content: [
      {
        type: 'image',
        source: {
          type: 'base64',
          media_type: 'image/png',
          data: imageBase64,
        },
      },
    ],
  }
  // Append toolResult to the next user message in your agent loop.
}

Screenshot scaling and coord-map

usecomputer screenshot always scales the output image so the longest edge is at most 1568 px. This keeps screenshots in a model-friendly size for computer-use agents.

Screenshot output includes:

• desktopIndex (display index used for capture)
• coordMap in the form captureX,captureY,captureWidth,captureHeight,imageWidth,imageHeight
• hint with usage text for coordinate mapping

Always pass the exact --coord-map value emitted by usecomputer screenshot to pointer commands when you are clicking coordinates from that screenshot. This maps screenshot-space coordinates back to real screen coordinates:

usecomputer screenshot ./shot.png --json
usecomputer click -x 400 -y 220 --coord-map "0,0,1600,900,1568,882"
usecomputer mouse move -x 100 -y 80 --coord-map "0,0,1600,900,1568,882"

To validate a target before clicking, use debug-point. It takes the same coordinates and --coord-map, captures a fresh full-desktop screenshot, and draws a red marker where the click would land. When --coord-map is present, it captures that same region so the overlay matches the screenshot you are targeting:

usecomputer debug-point -x 400 -y 220 --coord-map "0,0,1600,900,1568,882"

Kitty Graphics Protocol (agent-friendly screenshots)

When the AGENT_GRAPHICS environment variable contains kitty, the screenshot command emits the PNG image inline to stdout using the Kitty Graphics Protocol. This lets AI agents receive screenshots in a single tool call — no separate file read needed.

The protocol is supported by kitty-graphics-agent, an OpenCode plugin that intercepts Kitty Graphics escape sequences from CLI output and injects them as LLM-visible image attachments. To use it, add the plugin to your opencode.json:

{
  "plugin": ["kitty-graphics-agent"]
}

The plugin sets AGENT_GRAPHICS=kitty in the shell environment automatically. When the agent runs usecomputer screenshot, the image appears directly in the model's context window.

The JSON output includes "agentGraphics": true when the image was emitted inline, so programmatic consumers know the screenshot is already in context.

Keyboard commands

Type text

# Short text
usecomputer type "hello from usecomputer"

# Type from stdin (good for multiline or very long text)
cat ./notes.txt | usecomputer type --stdin --chunk-size 4000 --chunk-delay 15

# Simulate slower typing for apps that drop fast input
usecomputer type "hello" --delay 20

--delay is the per-character delay in milliseconds.

For very long text, prefer --stdin + --chunk-size so shell argument limits and app input buffers are less likely to cause dropped characters.

Press keys and shortcuts

# Single key
usecomputer press "enter"

# Chords
usecomputer press "cmd+s"
usecomputer press "cmd+shift+p"
usecomputer press "ctrl+s"

# Repeats
usecomputer press "down" --count 10 --delay 30

Modifier aliases: cmd/command/meta, ctrl/control, alt/option, shift, fn.

Platform note:

• macOS: cmd maps to Command.
• Windows/Linux: cmd maps to Win/Super.
• For app shortcuts that should work on Windows/Linux too, prefer ctrl+....

Coordinate options

Commands that target coordinates accept -x and -y flags:

• usecomputer click -x <n> -y <n>
• usecomputer hover -x <n> -y <n>
• usecomputer mouse move -x <n> -y <n>

mouse move is optional before click when click coordinates are already provided.

Legacy coordinate forms are also accepted where available.

Display index options

For commands that accept --display, the index is 0-based:

• 0 = first display
• 1 = second display
• 2 = third display

Example:

usecomputer screenshot ./shot.png --display 0 --json