AgentDesk

Desktops for AI agents   💻
Explore the docs »

View Demo · Report Bug · Request Feature

AgentDesk provides full-featured desktop environments which can be programatically controlled by AI agents. Spin them up locally or in the cloud.

▶ Built on agentd a runtime daemon which exposes a REST API for interacting with the desktop.

▶ Implements the DeviceBay Protocol.

Installation

pip install agentdesk

Quick Start

from agentdesk import Desktop

# Create a local VM
desktop = Desktop.local()

# Launch the UI for it
desktop.view(background=True)

# Open a browser to Google
desktop.open_url("https://google.com")

# Take actions on the desktop
desktop.move_mouse(500, 500)
desktop.click()
img = desktop.take_screenshot()

Usage

Create a local desktop

from agentdesk import Desktop

desktop = Desktop.local()

$ agentdesk create --provider qemu

*requires qemu

Create a remote desktop on GCE

desktop = Desktop.gce()

$ agentdesk create --provider gce

Create a remote desktop on EC2

desktop = Desktop.ec2()

$ agentdesk create --provider ec2

View the desktop in the UI

desktop.view()

$ agentdesk view old_mckinny

*requires docker

List desktops

Desktop.find()

$ agentdesk get

Delete a desktop

Desktop.delete("old_mckinny")

$ agentdesk delete old_mckinny

Use the desktop

desktop.open_url("https://google.com")

coords = desktop.mouse_coordinates()

desktop.move_mouse(500, 500)

desktop.click()

desktop.type_text("What kind of ducks are in Canada?")

desktop.press_key('Enter')

desktop.scroll()

img = desktop.take_screenshot()

Processors

Process images to make them more accessible to LMMs.

Grid

Add a coordinate grid on top of the image

from agentdesk.processors import GridProcessor

img = desktop.take_screenshot()

processor = GridProcessor()
grid_img = processor.process_b64(img)

Examples

Drawing Bot

See how to use a web-based drawing app with AgentDesk in our notebook.

GPT-4V

See how to use GPT-4V with AgentDesk in our notebook or agent.

Community

Come join us on Discord.

Developing

Please open an issue before creating a PR.

Changes to the VM happen in agentd.