Web Scout — MCP server for AI-driven web API & DOM discovery

[SanZiNEO]

Pre-submission Checklist

• This post follows the MCP community guidelines

What would you like to share?

What it does

A Python MCP server that helps AI autonomously discover web data sources:

• Opens a page → extracts full text as Markdown → AI picks keywords → auto-captures API requests
• Auto-falls back to DOM scanning when no API matches
• Outputs compressed field docs (12KB JSON → 800 chars, 15x compression) for AI to write scrapers from

How it's built

• Browser: DrissionPage ChromiumPage with tab.listen for real-time network monitoring
• API discovery: filters JSON responses → recursive field extraction → compression & dedup
• DOM discovery: JS bulk scanning (tab.run_js) → similarity merging → keyword-driven container search
• MCP framework: fastmcp, 12 tools
• Login detection + CAPTCHA handling (.nc_wrapper / text=安全验证)

Why

Existing fetch/seo/search MCP tools can retrieve pages but can't discover where the data lives. Web Scout bridges that gap — not a scraper, but a tool that tells AI how to scrape.

Relevant Links

• GitHub: https://github.com/SanZiNEO/web-scout
• 12 tools, MIT license, Python 3.10+, pip install -e .