Agent DevTools

Action-level visual debugging and task verification for Browser Use and Playwright agents.

Agent DevTools turns a browser-agent run into a local JSON trace and readable HTML timeline. It helps developers answer three concrete questions:

• What changed? Compare screenshots, URLs, and compact state before and after every recorded action.
• Why did it fail? Inspect operation errors, target diagnostics, browser errors, failed requests, and bounded evidence next to the responsible action.
• Did the task succeed? Keep action execution, optional action checks, and final task verification separate.

View a sample report

The included failure sample needs no API key, model, or running agent. All four browser actions execute successfully, but the agent opens the wrong product, so the final task check fails.

Download or inspect the generated sample report, then open report.html in a browser to explore its timeline and visual evidence. The committed sample is generated by examples/generate_sample_report.py using the same models and HTML generator as a real trace.

For a multi-run view, inspect the sample stability evaluation. It compares six deterministic attempts: three passes, two matching wrong-target failures, and one unverified early ending.

What it records

• Ordered navigate, click, fill, press, and scroll actions from supported integrations
• Arguments, timing, execution status, failure category, and failure reason
• Before-and-after screenshots, URLs, and compact structured state
• Action-level checks and final task verification as different result levels
• Controlled post-run replay for validated click and fill actions
• Repeated Playwright session replay with stable, intermittent, or unreproduced results, a concise first-difference summary, and automatic failed-action selection when no target number is supplied
• Agent-run exceptions shown as unverified with a sanitized exception type
• Browser page errors, console errors, failed requests, and HTTP error responses
• Versioned JSON sessions and a static HTML report after each observed run

Install and run with Browser Use

Agent DevTools is not yet published on PyPI. Install the early alpha directly from GitHub and install Chromium:

uv add "39-tools[browser-use] @ git+https://github.com/YYDongRo/39-tools.git"
uv run playwright install chromium

The displayed project name is Agent DevTools. The Python distribution is 39-tools, while imports use agent_devtools.

Keep your model provider key in the environment variable expected by Browser Use. Inside the application's async entry point, wrap an existing agent once:

from browser_use import Agent, Browser, ChatGoogle
from agent_devtools.browser_use import observe_browser_use_agent

task = "Open example.com and confirm the Example Domain page is open."
browser = Browser(headless=False)
raw_agent = Agent(
    task=task,
    llm=ChatGoogle(model="gemini-2.5-flash"),
    browser=browser,
    use_judge=True,
)
agent = observe_browser_use_agent(raw_agent)

await agent.run(max_steps=5)
agent.open_last_report()
await browser.stop()

The agent keeps its normal task and tools. The observer reads that task from the wrapped Agent, so you do not enter it a second time. It creates a unique local directory containing session.json, report.html, and per-action screenshots. Use agent.assert_last_task_passed() when failed or unverified tasks should fail a test. An explicit goal is still supported for agents that do not expose their task as agent.task.

One-time setup, then run tasks from the CLI

After installation and the one-time wrapper setup, a user can run the included CLI example, enter a task, and receive a report without adding recording code for that task:

cp agent_devtools.example.toml agent_devtools.toml
uv run --extra browser-use python examples/browser_use_cli.py \
  --headed --open-report

The command asks for the task, runs the existing Browser Use Agent, and prints the generated report.html path. It exits with a failure status when the Agent raises or the final task result is failed or unverified. The configuration file controls screenshots, redaction, summaries, report opening, and output directories; provider keys stay in environment variables.

For a custom desktop or browser Agent, keep your existing CLI and connect its run(task, *, tools=...) boundary to observe_agent(...) once. Agent DevTools records calls made through tools; it does not intercept arbitrary direct pyautogui or browser calls. See the CLI and custom-agent guide for the exact contract and a desktop integration example.

Optional configuration

To keep setup in one human-readable file, copy agent_devtools.example.toml to agent_devtools.toml, toggle the values you want, and pass it when wrapping:

agent = observe_browser_use_agent(raw_agent, config="agent_devtools.toml")

The file controls recording, screenshots, terminal summaries, automatic report opening, local trace output, and repeated-evaluation output. Credential-shaped metadata is redacted by default. Screenshots are not edited, so review them before sharing. The file does not contain the task or provider keys: the task remains on Agent(task=...), and credentials stay in the environment variables expected by Browser Use. Without config=..., the current recording behavior is unchanged apart from the default metadata redaction.

For a stronger final result without relying only on the Browser Use judge, add an optional deterministic check:

from agent_devtools.browser_use import (
    BrowserUseFinalStateCheck,
    observe_browser_use_agent,
)

agent = observe_browser_use_agent(
    raw_agent,
    final_check=BrowserUseFinalStateCheck(
        url_contains="/products/wireless-headphones",
        title_contains="Wireless Headphones",
    ),
)

The report uses these checks for the final result and keeps the model judge in collapsed evidence for comparison. If no check is provided, the existing judge behavior is unchanged. With use_judge=True, Browser Use already evaluates the same task with its model; Agent DevTools reuses that result instead of asking the developer to write a second goal or making another LLM request.

See the complete runnable example and Browser Use guide for setup, cleanup, output paths, and integration limits.

To see the debugger's failure case with a real Browser Use run, use examples/browser_use_failure.py. It opens the public example page successfully, then applies an intentionally impossible final title check. The report should show successful action execution beside a failed overall task result.

Evaluate repeated-run stability

Run the same Browser Use task sequentially with a fresh Agent each time:

from agent_devtools.browser_use import evaluate_browser_use_agent

evaluation = await evaluate_browser_use_agent(
    agent_factory=create_agent,
    task="Find the wireless headphones and open the correct product page.",
    runs=10,
    max_steps=15,
)
evaluation.open_report()

The aggregate report keeps every normal session trace, shows four distinct results (passed, failed, unverified, and errored), selects a representative successful trajectory, and groups repeated explainable failure patterns. The evaluator closes each Agent returned by the factory. See the stability evaluation guide for the factory contract, statistics, output layout, and limitations.

Run the provided workflow

To try the real local workflow, copy the human-readable configuration and set your Browser Use provider key in the shell (never in the TOML file):

cp agent_devtools.example.toml agent_devtools.toml
uv run --extra browser-use python examples/browser_use_evaluation.py \
  --runs 3 --open-report

The script creates a fresh agent for every attempt, writes one aggregate report under evaluations/browser-use/, and returns exit code 1 if any run is not explicitly passed. A failed evaluation still keeps all generated traces, so CI can save the report as an artifact. Add --headed to watch the browser. Python callers can use evaluation.assert_all_passed() for the same CI check. The complete workflow and configuration options are in the stability evaluation guide.

Understand the result

The report keeps three questions separate:

Result	Question	Meaning
Execution	Did the browser operation run?	The tool call completed or raised an error.
Action check	Did that action have its expected local effect?	Optional verification for one step.
Final check	Did the full trajectory satisfy the user request?	Determines the overall task result.

A click can execute successfully and still choose the wrong target. State changes are useful evidence, but they are not proof that the user's goal was completed.

Integration options

Integration	Best for	Entry point
Browser Use observer	Existing Browser Use 0.13.x agents	observe_browser_use_agent(...)
Playwright agent observer	Agents with run(user_request, *, tools=...)	observe_playwright_agent(...)
Generic agent observer	Agents with run(task, *, tools=...)	observe_agent(...)
Playwright tool wrapper	Existing browser tool objects	record_playwright_tools(...)
Generic sync/async wrapper	Framework-independent tool objects	record_tools(...), record_async_tools(...)
Core recorder	Building a custom adapter	SessionRecorder, record_action(...)

Detailed documentation:

• Browser Use integration
• Repeated-run stability evaluation
• Playwright and tool integrations
• CLI workflow and custom-agent integration
• Results, verification, and trace concepts
• Development and testing

The minimal agent boundary

If an agent exposes run(task, *, tools=...), wrap it once and let Agent DevTools inject the recording tools. When the task already lives on agent.task, you do not enter it again:

from agent_devtools import observe_agent

raw_agent = MyAgent(task="Open the settings page")
observed = observe_agent(raw_agent, my_tools, "trace/my-agent")
result = observed.run()
observed.open_last_report()

The wrapper records every callable method that the agent calls through the provided tool object. Optional capture_screenshot, observe_state, and task_verification callbacks add the same evidence used by the existing recorders. Async agents use observe_async_agent(...). This is an explicit adapter boundary, not a promise to intercept arbitrary direct desktop or browser calls.

To try this boundary with a real local browser and no model API key:

uv run --extra browser python examples/generic_agent_browser.py --headed

The deterministic demo opens examples/browser_click.html, records a navigation and click, verifies the final status text, and prints the report path. Remove --headed for a headless run.

Current scope and alpha status

Agent DevTools is an early alpha focused on Browser Use and Playwright agents. It records calls that pass through a supported observer or wrapped tool object; it cannot intercept arbitrary direct browser, desktop, or Android operations.

The Browser Use timeline intentionally omits read-only operations such as screenshots, extraction, state reads, and done, and it does not record hidden model reasoning.

Current limitations:

• Browser Use 0.13.x is the only official third-party agent adapter.
• Other agents require the Playwright/generic wrappers or a dedicated adapter.
• There is no hosted dashboard, general session replay, or automatic recovery.
• Async action recording is sequential.
• AI-assisted verification is optional, probabilistic, and not ground truth.

See PROJECT_PLAN.md for the current product boundary.

Privacy

Traces stay local by default, but they can contain URLs, typed arguments, page titles, screenshots, visible text, and bounded error details. Review and redact trace directories before sharing them. Provider keys are read from environment variables and are not written to reports.

Development

uv sync
uv run pytest
uv build

Browser tests and optional integrations use additional dependency groups. See the development guide for the complete commands.

Regenerate the sanitized sample report with:

uv run python examples/generate_sample_report.py
uv run python examples/generate_sample_evaluation.py

Contributing, security, and license

See CONTRIBUTING.md for development guidance and SECURITY.md for vulnerability reporting and trace-safety notes. Agent DevTools is available under the MIT License.