A Swift library for macOS automation — mouse, keyboard, screenshots, image recognition, and AI-powered agents.

This repository is inspired by pyautogui.

AI Agent that autonomously observes the screen and executes actions to achieve a goal.

sagui agent "Open Safari and search for Swift"

• macOS 27.0+
• Swift 6.2+

SwiftAutoGUI is available through Swift Package Manager.

in Package.swift add the following:

dependencies: [
    // Dependencies declare other packages that this package depends on.
    .package(url: "https://github.com/NakaokaRei/SwiftAutoGUI", branch: "master")
],
targets: [
    .target(
        name: "MyProject",
        dependencies: [..., "SwiftAutoGUI"]
    )
    ...
]

The sagui command-line tool is available via the Homebrew tap:

brew install NakaokaRei/tap/sagui

After install, grant Accessibility permission to your terminal in System Settings → Privacy & Security → Accessibility.

sagui --version
sagui key list
sagui key shortcut return
sagui mouse click --x 100 --y 200
sagui mouse click --right --x 100 --y 200

Quick start: Browser Agent guide

SwiftAutoGUIBrowser adds semantic automation for an existing Chrome, Edge, or other Chromium debugging session. It is a separate product, so applications that use only native macOS automation do not acquire browser-specific code.

Add the optional product to the client target:

.target(
    name: "MyProject",
    dependencies: [
        .product(name: "SwiftAutoGUI", package: "SwiftAutoGUI"),
        .product(name: "SwiftAutoGUIBrowser", package: "SwiftAutoGUI")
    ]
)

Start Chromium with a dedicated profile and a loopback debugging port. Chrome 136 and later do not honor remote debugging against the default profile.

"/Applications/Google Chrome for Testing.app/Contents/MacOS/Google Chrome for Testing" \
  --remote-debugging-port=9222 \
  --user-data-dir=/tmp/swift-auto-gui-browser-profile

Connect with an explicit navigation allowlist:

import SwiftAutoGUI
import SwiftAutoGUIBrowser

let browser = try await BrowserSession.connect(
    endpoint: URL(string: "http://127.0.0.1:9222")!,
    securityPolicy: BrowserSecurityPolicy(
        allowedDomains: ["example.com", "*.example.org"]
    )
)

let observation = try await browser.observe()
print(observation.formattedContext)

let apiKey = ProcessInfo.processInfo.environment["OPENAI_API_KEY"]!
let browserAgent = Agent(
    model: AutomationModels.openAI(apiKey: apiKey),
    automationBackend: browser
)

The browser Agent is intentionally browser-only: native app, window, AX-label, and coordinate mouse actions fail as unsupported instead of falling back to Accessibility or CGEvent. Stale DOM elements also fail safely without clicking a saved coordinate. Cross-origin navigation and downloads require a BrowserActionAuthorizing implementation; without one they are denied.

The sagui CLI supports deterministic CDP commands as well as the browser-only Agent:

sagui browser tabs
sagui browser observe --tab-id TARGET_ID
sagui browser click --tab-id TARGET_ID --role link --name "Issues" --domain github.com
sagui browser agent "Open issue 118" --domain github.com --allow-cross-origin

For complete API and module documentation, see the SwiftAutoGUI DocC site.

SwiftAutoGUI includes an Agent that can autonomously observe the screen, reason about what it sees, and execute actions in a loop until a goal is achieved. This follows the ReAct (Observe → Think → Act) pattern using a vision-capable LLM.

import SwiftAutoGUI

let backend = AutomationModels.openAI(apiKey: "sk-...", model: "gpt-5.6-sol")
let agent = Agent(
    model: backend,
    maxIterations: 15,
    visionMode: .automatic
)

let result = try await agent.run(goal: "Open Safari and search for Swift")
print("Completed: \(result.completed), Steps: \(result.iterationsUsed)")

The CLI defaults to the on-device Apple model. Choose a cloud provider explicitly:

sagui agent "Press the Save button" --vision-mode automatic
sagui agent "Open Safari and search for Swift" --provider openai
sagui browser agent "Open Issues" --domain github.com --provider openai

OpenAI reads OPENAI_API_KEY (or --api-key). --provider pcc requires Apple's managed entitlement in an eligible consuming app/executable; ordinary standalone CLI distribution is not assumed to qualify. --fallback-on-device opts into a local fallback. No cloud fallback is automatic.

The old --reasoning-effort option has been removed: the pinned Apple Chat Completions adapter does not forward it. Per-step Reasoning: output is a short user-facing explanation, not hidden chain-of-thought.

See the macOS 27 migration guide for API changes, context limits, provider compatibility, and PCC configuration.

When screen context is enabled, actionable Accessibility elements receive step-local identifiers such as [#12]. The agent can target these identifiers directly, resolves them again immediately before execution, and rejects stale elements safely. Each action returns a structured result describing the execution method, failure, UI change, and focus change. If an action changes the UI, the remaining batch is stopped and the agent observes the new state before continuing.

--vision-mode accepts always, automatic, or never. automatic omits the screenshot when actionable Accessibility elements are available; always preserves the original behavior and remains the default.

import SwiftAutoGUI

// Execute single actions
await Action.leftClick.execute()
await Action.write("Hello, World!").execute()
await Action.keyShortcut([.command, .a]).execute()  // Select all

// Build and execute action sequences
let actions: [Action] = [
    .move(to: CGPoint(x: 100, y: 100)),
    .wait(0.5),
    .leftClick,
    .write("Hello, SwiftAutoGUI!"),
    .keyShortcut([.returnKey])
]
await actions.execute()

SwiftAutoGUI ships as a Claude Code plugin so Claude can control native macOS applications and Chromium pages through the sagui CLI.

Inside Claude Code:

/plugin marketplace add NakaokaRei/SwiftAutoGUI
/plugin install swift-auto-gui@swift-auto-gui

This installs two skills:

• macos-control, invoked as /swift-auto-gui:macos-control, controls native macOS UI.
• browser-control, invoked as /swift-auto-gui:browser-control, controls Chromium pages through CDP without native input fallback.

The skills walk Claude through installing or updating the sagui binary when needed.

Grant the application running Claude Code (Terminal.app, iTerm, etc.) both:

• Accessibility — System Settings → Privacy & Security → Accessibility
• Screen Recording — System Settings → Privacy & Security → Screen Recording

These permissions are required for macos-control; browser-only CDP actions do not require them.

For full details, see the macos-control and browser-control skill definitions.

• NakaokaRei

MIT license. See the LICENSE file for details.