Repository navigation
how to build a browser agent
The loop is genuinely simple, and people who build one are usually surprised twice: first that it works in an afternoon, then that the afternoon version is about ten percent of the job. This page is both halves.
Four steps, repeated:
- Describe the page to the model. A structured description, not raw HTML: the accessibility tree, or the interactive elements with stable identifiers. Raw HTML is enormous, mostly irrelevant, and will exhaust your context on the third page.
- Ask for one action. Constrained to a small vocabulary: navigate, click, type, select, press, read, screenshot, done. For calibration, our own production server settled on 17 tools totalling 9,207 characters of description, 3,497 tokens with their argument schemas (measured 2026-10-12) covering browsers, reading, the pointer, the keyboard, a live view and a JavaScript reader. If your vocabulary is much larger than that, the model is choosing between overlapping verbs.
- Perform it, and get the resulting state.
- Decide whether to stop.
That is it. The interesting part is that each of these four has a failure mode that only appears after the demo works.
Describing the page well. Too much and you pay for tokens and the model gets lost; too little and it clicks the wrong thing. The working shape is: interactive elements only, with a short label, a role and a stable reference the next call can use. If your descriptions contain CSS selectors you invented, the model will invent them too, and they will be wrong after the next deploy.
Making the action reach the page correctly. Any automation library can click a selector. Two things bite later: elements that no selector describes, which need a coordinate path, and the fact that a click issued from script is not the same event as a click issued from a pointer. Whether that matters depends entirely on the site.
Knowing when to stop. The failure that costs the most money. Without an explicit stopping condition, an agent that has the answer will keep browsing. Build the stop into the loop, not into the prompt alone: a turn budget, and a required "done" action with the result attached.
Recovering from surprise. A dialog, a cookie banner, a redirect to a login, an empty result. The demo never hits these and the real task hits one within five pages. Most of the engineering in a mature agent is here.
Building the loop yourself is a good way to understand the problem and a poor way to end up with something maintainable. The parts worth taking off the shelf:
The page description. MCP servers already produce a model-friendly snapshot with stable references. Reimplementing that well is weeks.
The action vocabulary. Same reason. And a server gives you the coordinate fallback that you would otherwise discover you needed.
The whole loop, if you do not need it to be yours. browser-use, Skyvern and Stagehand are the agent layer, open source, with the recovery cases already handled. Open-source agentic browsers lays out what each gives you.
The cheapest starting point of all is to not build a program: register a browser MCP server with an assistant you already run and prompt it. You get the loop, the model and the interface for the cost of a config block, and you find out in an hour whether the task is even feasible. The MCP server has ours, and choosing an MCP server covers the field.
The context fills up. Thirty snapshots of a page in one conversation. Keep the last state and a summary of the path, not the whole history.
It loops. Click, page does not change, click again, forever. Detect that the state did not change and force a different action or a stop.
It cannot see what it needs. A canvas, a map, a widget with no accessible name. This is the point where you need screenshots and coordinates, and adding them later is harder than starting with the path in place.
The site starts refusing it. Not an agent problem, and switching model or framework will not touch it. Why an agent gets blocked has the attribution: usually the address, the pacing, or a stock automation build, in that order.
How hard is it to build a browser agent? The loop is an afternoon. The recovery cases are the project.
Should I use raw HTML or the accessibility tree? The tree, or a reduced element list. Raw HTML will not fit and mostly is not about anything the agent can act on.
Do I need a vision model? Only where the structured description fails. Build the coordinate path anyway; you will need it.
Should I build on Playwright or Selenium? Either drives a page. Neither changes what the page can observe about the browser, which is a separate layer.
Can I just use an MCP server instead? Yes, and it is the right first move: you find out whether the task is feasible before writing a program.
See also: writing tasks for a browser agent, which LLM for browser automation, and how the tools are shaped for why a small action vocabulary beats a large one.
- microsoft/playwright-mcp, retrieved 2026-09-10, for the snapshot-with-references design that this page recommends copying.
- The browser-use and Skyvern repositories for the off-the-shelf agent layer.
Written while maintaining a browser MCP server, which is the "take it off the shelf" recommendation in the middle of the page. The loop is described in full anyway, because knowing what is inside it is how you tell whether a tool is doing it well.
- OpenAI Operator alternatives
- Open-source Operator-style agents
- Is OpenAI Operator still available?
- OpenAI Operator vs Claude computer use
- browser-use alternatives
- Choosing an AI browser agent
- Open-source AI browser agents
- Open-source computer-use agents
- What is an AI web agent?
- AI browser agents vs traditional scraping
- Cloud browser infrastructure for AI agents, explained
- Browserbase alternatives
- Firecrawl vs an AI browser agent
- Skyvern alternatives
- Stagehand vs browser-use
- Project Mariner is gone: what replaced it
- Manus alternatives
- Gemini computer use vs Claude computer use
- A stealth browser MCP, reviewed honestly by its own wiki
- AI browser vs AI browser agent: which one do you want?
- AI browser agent vs RPA: which one fits the job
- AI browser agent vs n8n, Zapier and Make
- Vercel agent-browser alternatives, compared honestly
- What is an agentic browser? Definition and the two kinds
- Open-source agentic browsers: the three layers, compared
- Choosing an MCP server for browser automation: four axes
- Stealth MCP servers compared: Camoufox, nodriver, Patchright
- Playwright MCP alternatives, and the three you don't need
- Autonomous browser agents: the four rungs of autonomy
- What is actually free in the AI browser agent stack
- browser-use on GitHub: what the repo actually gives you
- Playwright MCP vs Chrome DevTools MCP: different jobs
- How to choose among MCP servers: a map by category
- Which MCP servers are worth adding to Claude Code
- MCP on GitHub: finding servers and judging them fast
- MCP vs an API: the decision, and what the wrapper costs
- MCP alternatives: when the protocol is the wrong shape
- Why does my AI agent get blocked?
- The timing signal AI agents give off
- Agent retry loops trip rate limits, not fingerprints
- Claude computer use detected as a bot
- browser-use getting blocked: what you can and cannot change
- Playwright MCP session blocked: four causes, four fixes
- Playwright MCP and captchas: what actually gets you past
- Cloudflare and a browser MCP server: what is being read
- Can an AI agent solve a captcha? The honest answer
- Getting an AI agent to fill out forms
- The best model for an MCP browser agent, and what it really costs
- Browser problem or model problem?
- How to use a Playwright MCP server with Claude Code
- Extracting data to a CSV with an AI agent
- Monitoring a page for changes with an AI agent
- How to let Claude Desktop control a browser
- How to add a browser to Cursor as an MCP server
- Using an AI agent to hunt for apartments
- Getting website data into Google Sheets with an AI agent
- Using an AI agent to download invoices from portals
- AI agents for web research
- Using an AI agent to test your own website
- How to add a browser to Cline as an MCP server
- Posting to social media with an AI agent
- Posting to Facebook with an AI agent
- Posting to Instagram with an AI agent
- Posting to X with an AI agent
- Automating LinkedIn posts: read this first
- Appointment bots: what they are and what an agent can legitimately do
- Track prices across sites with an AI agent
- Build a lead list with an AI browser agent
- Run an AI browser agent on a schedule
- AI browser agent with a local LLM: what changes
- Should you log your AI agent into your accounts?
- How to write a task an AI browser agent can follow
- Move data between two web apps with an AI agent
- The MCP server
- How the tools are shaped, and why
- Playwright MCP vs the Playwright CLI: which fits when
- Playwright MCP: browser is already in use, and the fix
- Playwright MCP best practices: four decisions that matter
- Playwright MCP with a proxy, and the three leaks it leaves
- A browser MCP server in GitHub Copilot: setup and limits
- Using a browser MCP server for web scraping: the pattern
- Which LLM for browser automation: the four properties
- How to build a browser agent, and what to take instead
- Getting an AI agent to log into a website: three routes
- MCP tools, resources and prompts: who controls each
- How many MCP tools is too many? The context arithmetic
- How to build an MCP server: the decisions, not the scaffold
- Local or remote MCP server: what changes, and what does not
- Writing an MCP client in Python: the thirty-line version
- Self-hosted AI agent: what one actually costs to run
- How long an AI browser agent takes per step, measured
- Text, HTML, snapshot or screenshot: what the agent should read
- Giving an AI browser agent a stopping condition
- Keeping an AI browser agent out of destructive actions
- Why did the AI agent click the wrong thing
- When the page changes under the AI agent
- Running one AI agent task across a list of sites
- Seeing a page as it appears in another country
- Getting data out of a dashboard with no export button
- Two browsers in one session: main and support
- Finding the dead links on a site with an AI agent
- Filling a CRM record from a company's website
- One form submission per spreadsheet row, with an AI agent
- Dated screenshots of a page as evidence
- Checking order and delivery status with an AI agent
- Reading a PDF that opens inside the browser
- Summarising a long page or thread with an AI agent
- Collecting every image on a page with its caption
- Collecting event and course listings with an AI agent
- Cancelling a subscription with an AI agent
- What an AI agent can and cannot do inside an iframe
- Shadow DOM and an AI agent: you can click it, you cannot read it
- How to add a browser to Codex as an MCP server
- What a page snapshot costs, per control
- How to let Gemini CLI use a browser
- Native selects and the ones that only look like selects
- Clicking by selector or by coordinates
- How long the agent waits before it gives up
- What a second browser costs
- Uploading a file with an AI agent, and why this one cannot
- Watching the agent work, and when it is worth it
- When not to use an AI browser agent
- Agent or script: deciding once instead of every time
- Using the keyboard instead of the mouse
- Secrets in an agent task: where they end up
- What an agent run should log
- Deduplicating what an AI agent collects
- Normalising values across sites
- Validating an AI agent's output
- Reading a table with an AI agent
- Driving a site's own search and filters
- The task works headed and fails headless
- How to let the AdaL CLI use a browser