Web scraper de alto rendimiento con arquitectura modular para datasets RAG, crawling inteligente y exportación multi-formato.
Quick Start · Architecture · Features · CLI Reference · MCP Server · Developer Guide
# Install
git clone https://github.com/XaviCode1000/webfang.git
cd webfang
cargo install --path crates/webfang_cli
# Scrape a single page
webfang --url https://example.com
# Crawl an entire site
webfang --url https://example.com --use-sitemap --max-pages 50
# Export for RAG pipelines
webfang --url https://example.com --export-format jsonl --clean-aiOutput is saved to output/ as Markdown by default.
Clean Architecture with enforced dependency direction across 5 workspace crates:
webfang_cli ──→ webfang_tui ──→ webfang_core ←── webfang_ai
webfang_cli ──→ webfang_mcp ──→ webfang_core
webfang_cli ──────────────────────→ webfang_core
| Crate | Purpose | Key Dependencies |
|---|---|---|
webfang_core |
Domain, application, infrastructure | wreq, tokio, scraper, lol_html |
webfang_ai |
ONNX semantic cleaning | tract-onnx |
webfang_tui |
Terminal UI | ratatui |
webfang_mcp |
MCP server for AI agents | rmcp |
webfang_cli |
Binary entry point + CLI parsing | clap |
Dependency direction: CLI → {TUI, MCP, AI} → Core. No circular dependencies.
| Feature | Description |
|---|---|
| Content extraction | Readability-based extraction — strips menus, ads, sidebars |
| AI semantic cleaning | ONNX embeddings filter irrelevant content (feature ai) |
| Multi-format export | Markdown, JSON, JSONL (RAG), Vector (embeddings) |
| Obsidian integration | Direct vault saves with wiki-links and metadata |
| Sitemap discovery | Auto-discovers all pages via robots.txt + sitemap.xml |
| Asset download | Images and documents (PDF, DOCX, XLSX) |
| WAF detection | Detects Cloudflare, reCAPTCHA, hCaptcha, DataDome |
| MCP server | 34+ tools for AI agent integration |
| Rate limiting | Configurable with Retry-After respect |
| Resume | Continues interrupted crawls with --resume |
| TLS fingerprinting | wreq impersonates real browsers to bypass WAFs |
| TUI selector | Interactive URL selection with ratatui |
# Single page
webfang --url https://example.com
# With selector (CSS)
webfang --url https://example.com --selector "article h1"
# Multi-page crawl
webfang --url https://example.com --max-pages 50 --concurrency 4
# Sitemap-based crawl
webfang --url https://example.com --use-sitemap --sitemap-url https://example.com/sitemap.xmlwebfang --url https://example.com --format markdown # Default
webfang --url https://example.com --format json
webfang --url https://example.com --export-format jsonl
webfang --url https://example.com --export-format vectorwebfang --url https://example.com --clean-ai --export-format jsonlwebfang --url https://example.com --obsidian-wiki-links --quick-savewebfang --url https://example.com --max-pages 100 --delay-ms 1000 --timeout-secs 30
webfang --url https://example.com --download-images --download-documents
webfang --url https://example.com --dry-run
webfang --url https://example.com --quietwebfang --helpThe MCP server provides 34+ tools for AI agent integration:
# stdio mode (for OpenCode, Claude Desktop, Cursor)
cargo run -p webfang_mcp --example mcp_server --quiet
# HTTP mode
cargo run -p webfang_mcp --example mcp_server| Category | Tools |
|---|---|
| Scraping | scrape_url, scrape_batch, crawl_site, crawl_with_sitemap |
| Content | clean_html, extract_links, convert_html_to_markdown |
| WAF | detect_waf, verify_waf_integrity, list_waf_providers |
| Export | export_file, export_jsonl, export_vector |
| Obsidian | detect_obsidian_vault, search_obsidian, build_obsidian_uri |
| URLs | validate_url, normalize_url, is_internal_link |
Config file: ~/.config/webfang/config.toml
format = "markdown"
max_pages = 50
delay_ms = 500
use_sitemap = trueCLI arguments override config file values.
| Feature | Activates | Install |
|---|---|---|
default |
images + documents | cargo install --path crates/webfang_cli |
ai |
Semantic cleaning with ONNX (~90MB model) | --features ai |
ui |
Interactive TUI with ratatui | --features ui |
mcp |
MCP server for AI agents | --features mcp |
persistence |
SQLite checkpoint store | --features persistence |
otel |
OpenTelemetry observability | --features otel |
console |
Tokio console (debugging) | --features console |
# Run all tests
cargo nextest run --workspace
# Run with coverage
cargo llvm-cov --all-features
# Run Miri (memory safety verification)
cargo +nightly miri test --libTest suite: 1,337 tests across unit, integration, and behavioral layers.
Miri status: Domain + Core layers verified for Undefined Behavior. Infrastructure layer partially verified (servo_arc/btls FFI limitations documented).
webfang/
├── crates/
│ ├── webfang_core/ # Domain + application + infrastructure
│ ├── webfang_ai/ # AI/ONNX inference
│ ├── webfang_tui/ # Terminal UI
│ ├── webfang_mcp/ # MCP server
│ └── webfang_cli/ # Binary entry point
├── Cargo.toml # Workspace manifest
└── .github/workflows/ci.yml # CI pipeline
# Quick verification (check + clippy + fmt)
cargo check --workspace && cargo clippy --workspace -- -D warnings && cargo fmt --all -- --check
# Run tests
cargo nextest run --workspace
# Build release
cargo build --release -p webfang_cli
# Build with all features
cargo build --release -p webfang_cli --features full
# Re-index GitNexus (code intelligence)
gitnexus analyze --index-only --skip-agents-md- Dependency direction: CLI → {TUI, MCP, AI} → Core (never reverse)
- Port/Adapter pattern: Domain defines traits, Infrastructure implements them
- Error types: DomainError, InfraError, AppError → ScraperError (dual wrapping)
- User-facing errors: Spanish. Internal logs: English.
Stack: Rust 1.88 · Tokio · wreq (TLS fingerprint) · ratatui · scraper 0.27 · lol_html · tract-onnx
| Resource | Covers |
|---|---|
| AGENTS.md | AI agent instructions, GitNexus integration |
| Wiki | Architecture, API reference, guides |
webfang --help |
Full CLI reference |
- Fork → branch
feature/name→ commit → PR - Tests must pass:
cargo nextest run --workspace - Conventional Commits:
feat:,fix:,refactor:,ci:,docs: - Read AGENTS.md for architecture rules and GitNexus usage
MIT OR Apache-2.0