webfang

Web scraper de alto rendimiento con arquitectura modular para datasets RAG, crawling inteligente y exportación multi-formato.

Quick Start · Architecture · Features · CLI Reference · MCP Server · Developer Guide

---

Quick Start

# Install
git clone https://github.com/XaviCode1000/webfang.git
cd webfang
cargo install --path crates/webfang_cli

# Scrape a single page
webfang --url https://example.com

# Crawl an entire site
webfang --url https://example.com --use-sitemap --max-pages 50

# Export for RAG pipelines
webfang --url https://example.com --export-format jsonl --clean-ai

Output is saved to output/ as Markdown by default.

---

Architecture

Clean Architecture with enforced dependency direction across 5 workspace crates:

webfang_cli ──→ webfang_tui ──→ webfang_core ←── webfang_ai
webfang_cli ──→ webfang_mcp ──→ webfang_core
webfang_cli ──────────────────────→ webfang_core

Crate	Purpose	Key Dependencies
webfang_core	Domain, application, infrastructure	wreq, tokio, scraper, lol_html
webfang_ai	ONNX semantic cleaning	tract-onnx
webfang_tui	Terminal UI	ratatui
webfang_mcp	MCP server for AI agents	rmcp
webfang_cli	Binary entry point + CLI parsing	clap

Dependency direction: CLI → {TUI, MCP, AI} → Core. No circular dependencies.

---

Features

Feature	Description
Content extraction	Readability-based extraction — strips menus, ads, sidebars
AI semantic cleaning	ONNX embeddings filter irrelevant content (feature ai)
Multi-format export	Markdown, JSON, JSONL (RAG), Vector (embeddings)
Obsidian integration	Direct vault saves with wiki-links and metadata
Sitemap discovery	Auto-discovers all pages via robots.txt + sitemap.xml
Asset download	Images and documents (PDF, DOCX, XLSX)
WAF detection	Detects Cloudflare, reCAPTCHA, hCaptcha, DataDome
MCP server	34+ tools for AI agent integration
Rate limiting	Configurable with Retry-After respect
Resume	Continues interrupted crawls with --resume
TLS fingerprinting	wreq impersonates real browsers to bypass WAFs
TUI selector	Interactive URL selection with ratatui

---

CLI Reference

Basic usage

# Single page
webfang --url https://example.com

# With selector (CSS)
webfang --url https://example.com --selector "article h1"

# Multi-page crawl
webfang --url https://example.com --max-pages 50 --concurrency 4

# Sitemap-based crawl
webfang --url https://example.com --use-sitemap --sitemap-url https://example.com/sitemap.xml

Output formats

webfang --url https://example.com --format markdown    # Default
webfang --url https://example.com --format json
webfang --url https://example.com --export-format jsonl
webfang --url https://example.com --export-format vector

AI cleaning

webfang --url https://example.com --clean-ai --export-format jsonl

Obsidian

webfang --url https://example.com --obsidian-wiki-links --quick-save

Control

webfang --url https://example.com --max-pages 100 --delay-ms 1000 --timeout-secs 30
webfang --url https://example.com --download-images --download-documents
webfang --url https://example.com --dry-run
webfang --url https://example.com --quiet

Full reference

webfang --help

---

MCP Server

The MCP server provides 34+ tools for AI agent integration:

# stdio mode (for OpenCode, Claude Desktop, Cursor)
cargo run -p webfang_mcp --example mcp_server --quiet

# HTTP mode
cargo run -p webfang_mcp --example mcp_server

Category	Tools
Scraping	scrape_url, scrape_batch, crawl_site, crawl_with_sitemap
Content	clean_html, extract_links, convert_html_to_markdown
WAF	detect_waf, verify_waf_integrity, list_waf_providers
Export	export_file, export_jsonl, export_vector
Obsidian	detect_obsidian_vault, search_obsidian, build_obsidian_uri
URLs	validate_url, normalize_url, is_internal_link

---

Configuration

Config file: ~/.config/webfang/config.toml

format = "markdown"
max_pages = 50
delay_ms = 500
use_sitemap = true

CLI arguments override config file values.

---

Build Features

Feature	Activates	Install
default	images + documents	cargo install --path crates/webfang_cli
ai	Semantic cleaning with ONNX (~90MB model)	--features ai
ui	Interactive TUI with ratatui	--features ui
mcp	MCP server for AI agents	--features mcp
persistence	SQLite checkpoint store	--features persistence
otel	OpenTelemetry observability	--features otel
console	Tokio console (debugging)	--features console

---

Testing

# Run all tests
cargo nextest run --workspace

# Run with coverage
cargo llvm-cov --all-features

# Run Miri (memory safety verification)
cargo +nightly miri test --lib

Test suite: 1,337 tests across unit, integration, and behavioral layers.

Miri status: Domain + Core layers verified for Undefined Behavior. Infrastructure layer partially verified (servo_arc/btls FFI limitations documented).

---

Developer Guide

Workspace structure

webfang/
├── crates/
│   ├── webfang_core/     # Domain + application + infrastructure
│   ├── webfang_ai/       # AI/ONNX inference
│   ├── webfang_tui/      # Terminal UI
│   ├── webfang_mcp/      # MCP server
│   └── webfang_cli/      # Binary entry point
├── Cargo.toml                 # Workspace manifest
└── .github/workflows/ci.yml  # CI pipeline

Development commands

# Quick verification (check + clippy + fmt)
cargo check --workspace && cargo clippy --workspace -- -D warnings && cargo fmt --all -- --check

# Run tests
cargo nextest run --workspace

# Build release
cargo build --release -p webfang_cli

# Build with all features
cargo build --release -p webfang_cli --features full

# Re-index GitNexus (code intelligence)
gitnexus analyze --index-only --skip-agents-md

Architecture rules

• Dependency direction: CLI → {TUI, MCP, AI} → Core (never reverse)
• Port/Adapter pattern: Domain defines traits, Infrastructure implements them
• Error types: DomainError, InfraError, AppError → ScraperError (dual wrapping)
• User-facing errors: Spanish. Internal logs: English.

Stack: Rust 1.88 · Tokio · wreq (TLS fingerprint) · ratatui · scraper 0.27 · lol_html · tract-onnx

---

Documentation

Resource	Covers
AGENTS.md	AI agent instructions, GitNexus integration
Wiki	Architecture, API reference, guides
webfang --help	Full CLI reference

---

Contributing

1. Fork → branch feature/name → commit → PR
2. Tests must pass: cargo nextest run --workspace
3. Conventional Commits: feat:, fix:, refactor:, ci:, docs:
4. Read AGENTS.md for architecture rules and GitNexus usage

---

License

MIT OR Apache-2.0