Repository navigation
mcp for web scraping
The honest version, first: a browser MCP server is excellent at finding out how to scrape a site and bad at doing the scraping. Every step is a round trip through a model. That is a few seconds and a few thousand tokens per page. A plain script does the same page in a fraction of a second for nothing.
The floor under that estimate is measurable and it is ours: our server's 17 tool definitions are 3,497 tokens resent on every turn (9,207 characters of description plus each tool's argument schema, counted with a tokenizer on 2026-10-12). Page content and the model's own reasoning stack on top of that floor. Multiply by turns, then by pages, and phase 2 below stops being a style preference.
If the data is already in the HTML the server sends, the browser was never the question and neither was the protocol: MCP against a plain API and against RAG is the comparison to read before this page's pattern, not after it.
Which does not make MCP useless here. It makes it useful in a specific place, and the pattern below is what people converge on after they have paid for the naive version once.
Phase 1, with the model. Open the site. Ask the agent to find the listing, the pagination control, the fields you want and the shape of the detail page. Have it report the selectors it used, not just the values it found. This is one session, maybe twenty turns, and it replaces the half hour you would have spent in devtools.
Phase 2, without the model. Write those selectors into ordinary code and let it run over ten thousand pages. Deterministic, fast, free, and testable.
Phase 3, back to the model, on failure only. When the script starts returning empty fields, that is the signal that the page changed. Reopen the MCP session, ask what the page looks like now, update the selectors. The model is handling the exception, which is the thing it is good at, rather than the routine, which it is not.
The trap this avoids is real and expensive: a model rediscovering the same layout on every one of ten thousand pages, slowly, and with the possibility of deciding differently on page 4,000 than on page 3,999. A scrape that is not reproducible is not a dataset.
When every page is shaped differently. A hundred suppliers' sites with the same information in a hundred layouts is precisely the job a selector cannot express and a model can. The cost per page is high and there is no alternative that is lower, because the alternative is a hundred scripts.
When getting to the data is a flow. Search, filter, expand, accept, then read. If the sequence varies by page and cannot be replayed blindly, keeping the model in the loop is cheaper than encoding every branch.
Both are the same underlying rule: the model is worth its cost exactly where the next step is not knowable in advance. Everywhere else it is an expensive way to run code you could have written.
Three properties matter for extraction specifically, and tool count does not.
Reading without scripting. A text read and a structured snapshot, so the model can look at the page without you writing an evaluation for it. Microsoft's server returns an accessibility tree with stable references, which is the right shape for this.
A path to what selectors cannot reach. Coordinates, for a canvas or a custom widget. Otherwise phase 1 stalls on exactly the pages you needed help with.
A JavaScript reader. For the remainder: a value in a data attribute, a JSON blob in a script tag, the thing that never made it into the DOM as text.
How the tools are shaped, and why argues that in general; choosing an MCP server applies it to the field.
Volume scraping fails for reasons that have nothing to do with which server you picked. Rate limits, an address with a datacenter reputation, and pacing that no person could produce. A model in the loop actually makes the pacing question worse in one direction and better in another: slower per page, but with an irregularity that is accidental rather than designed.
Before concluding your server is the problem, read why an agent gets blocked, and if you are choosing between an agent and a scraper at all, AI browser agents vs traditional scraping is the wider comparison.
And the boundary this project keeps: nothing here defeats a site's protections, and a page that is served only behind a challenge stays behind it. Can an AI agent solve a captcha is the direct answer.
Can I scrape with Playwright MCP? Yes, and you should use it to work out how to scrape rather than to do the scraping.
Is an MCP server better than BeautifulSoup? Different jobs. If the data is in the HTML the server sent, you never needed a browser at all.
How much does it cost per page? Dominated by model tokens: a snapshot plus a few actions per page, every page. That is the number that makes phase 2 obvious.
Can the model extract to CSV? It can produce the rows. Have it produce the selectors too, so the next ten thousand rows do not cost the same.
Does an MCP server get past anti-bot protection? No, and the servers that say so are making a claim rather than a statement.
See also: Playwright MCP best practices, getting data into a spreadsheet with an agent, and Playwright MCP vs the CLI.
- microsoft/playwright-mcp, retrieved 2026-09-10, for the accessibility-tree snapshot and the vision capability.
Written while maintaining an MCP browser server. The advice that costs us the most is phase 2, and it is the middle of the page rather than a footnote because it is the part that saves the reader real money.
- OpenAI Operator alternatives
- Open-source Operator-style agents
- Is OpenAI Operator still available?
- OpenAI Operator vs Claude computer use
- browser-use alternatives
- Choosing an AI browser agent
- Open-source AI browser agents
- Open-source computer-use agents
- What is an AI web agent?
- AI browser agents vs traditional scraping
- Cloud browser infrastructure for AI agents, explained
- Browserbase alternatives
- Firecrawl vs an AI browser agent
- Skyvern alternatives
- Stagehand vs browser-use
- Project Mariner is gone: what replaced it
- Manus alternatives
- Gemini computer use vs Claude computer use
- A stealth browser MCP, reviewed honestly by its own wiki
- AI browser vs AI browser agent: which one do you want?
- AI browser agent vs RPA: which one fits the job
- AI browser agent vs n8n, Zapier and Make
- Vercel agent-browser alternatives, compared honestly
- What is an agentic browser? Definition and the two kinds
- Open-source agentic browsers: the three layers, compared
- Choosing an MCP server for browser automation: four axes
- Stealth MCP servers compared: Camoufox, nodriver, Patchright
- Playwright MCP alternatives, and the three you don't need
- Autonomous browser agents: the four rungs of autonomy
- What is actually free in the AI browser agent stack
- browser-use on GitHub: what the repo actually gives you
- Playwright MCP vs Chrome DevTools MCP: different jobs
- How to choose among MCP servers: a map by category
- Which MCP servers are worth adding to Claude Code
- MCP on GitHub: finding servers and judging them fast
- MCP vs an API: the decision, and what the wrapper costs
- MCP alternatives: when the protocol is the wrong shape
- Why does my AI agent get blocked?
- The timing signal AI agents give off
- Agent retry loops trip rate limits, not fingerprints
- Claude computer use detected as a bot
- browser-use getting blocked: what you can and cannot change
- Playwright MCP session blocked: four causes, four fixes
- Playwright MCP and captchas: what actually gets you past
- Cloudflare and a browser MCP server: what is being read
- Can an AI agent solve a captcha? The honest answer
- Getting an AI agent to fill out forms
- The best model for an MCP browser agent, and what it really costs
- Browser problem or model problem?
- How to use a Playwright MCP server with Claude Code
- Extracting data to a CSV with an AI agent
- Monitoring a page for changes with an AI agent
- How to let Claude Desktop control a browser
- How to add a browser to Cursor as an MCP server
- Using an AI agent to hunt for apartments
- Getting website data into Google Sheets with an AI agent
- Using an AI agent to download invoices from portals
- AI agents for web research
- Using an AI agent to test your own website
- How to add a browser to Cline as an MCP server
- Posting to social media with an AI agent
- Posting to Facebook with an AI agent
- Posting to Instagram with an AI agent
- Posting to X with an AI agent
- Automating LinkedIn posts: read this first
- Appointment bots: what they are and what an agent can legitimately do
- Track prices across sites with an AI agent
- Build a lead list with an AI browser agent
- Run an AI browser agent on a schedule
- AI browser agent with a local LLM: what changes
- Should you log your AI agent into your accounts?
- How to write a task an AI browser agent can follow
- Move data between two web apps with an AI agent
- The MCP server
- How the tools are shaped, and why
- Playwright MCP vs the Playwright CLI: which fits when
- Playwright MCP: browser is already in use, and the fix
- Playwright MCP best practices: four decisions that matter
- Playwright MCP with a proxy, and the three leaks it leaves
- A browser MCP server in GitHub Copilot: setup and limits
- Using a browser MCP server for web scraping: the pattern
- Which LLM for browser automation: the four properties
- How to build a browser agent, and what to take instead
- Getting an AI agent to log into a website: three routes
- MCP tools, resources and prompts: who controls each
- How many MCP tools is too many? The context arithmetic
- How to build an MCP server: the decisions, not the scaffold
- Local or remote MCP server: what changes, and what does not
- Writing an MCP client in Python: the thirty-line version
- Self-hosted AI agent: what one actually costs to run
- How long an AI browser agent takes per step, measured
- Text, HTML, snapshot or screenshot: what the agent should read
- Giving an AI browser agent a stopping condition
- Keeping an AI browser agent out of destructive actions
- Why did the AI agent click the wrong thing
- When the page changes under the AI agent
- Running one AI agent task across a list of sites
- Seeing a page as it appears in another country
- Getting data out of a dashboard with no export button
- Two browsers in one session: main and support
- Finding the dead links on a site with an AI agent
- Filling a CRM record from a company's website
- One form submission per spreadsheet row, with an AI agent
- Dated screenshots of a page as evidence
- Checking order and delivery status with an AI agent
- Reading a PDF that opens inside the browser
- Summarising a long page or thread with an AI agent
- Collecting every image on a page with its caption
- Collecting event and course listings with an AI agent
- Cancelling a subscription with an AI agent
- What an AI agent can and cannot do inside an iframe
- Shadow DOM and an AI agent: you can click it, you cannot read it
- How to add a browser to Codex as an MCP server
- What a page snapshot costs, per control
- How to let Gemini CLI use a browser
- Native selects and the ones that only look like selects
- Clicking by selector or by coordinates
- How long the agent waits before it gives up
- What a second browser costs
- Uploading a file with an AI agent, and why this one cannot
- Watching the agent work, and when it is worth it
- When not to use an AI browser agent
- Agent or script: deciding once instead of every time
- Using the keyboard instead of the mouse
- Secrets in an agent task: where they end up
- What an agent run should log
- Deduplicating what an AI agent collects
- Normalising values across sites
- Validating an AI agent's output
- Reading a table with an AI agent
- Driving a site's own search and filters
- The task works headed and fails headless
- How to let the AdaL CLI use a browser