Generic BFS Documentation Crawler

Reusable, configurable full-site crawler for online documentation.

Refined from a real-world documentation crawl. Zero hardcoded site logic — everything is configurable via YAML or dict API.

---

Core Capabilities

Feature	Description
BFS Recursive Crawl	Discover all pages from seed URLs, layer by layer
JS Rendering	Playwright-driven browser engine, never miss JS-generated sidebar links
Static Mode	Also supports plain requests mode for static HTML sites
Content Cleaning	trafilatura → CSS selectors → smart body extraction (3-layer fallback)
SQLite + FTS5	Auto-create tables, full-text search index, triggers
Resume / Pause	State auto-saved every 100 pages, restart picks up where you left off
Quality Marking	Short pages auto-tagged low, can be re-extracted on next run
Module Categorization	Auto-group pages by URL path structure
Pure Python	Install deps, run — no external runtime

---

Quick Start

1. Install

# Python dependencies
pip install -r requirements.txt

# Playwright browser (first time only, skip if using requests mode)
playwright install chromium

2. Create Config

Copy example_config.yaml and edit for your target site:

base_url: "https://your-docs.example.com"
seed_urls:
  - "https://your-docs.example.com/index.html"
url_filters:
  path_prefix: "/docs/"
  allowed_extensions: [".htm", ".html"]
db_path: "my_docs.db"
target_language: "en"
render_engine: "playwright"   # or "requests" for static sites

3. Run — Three Ways

Method	Best For
① YAML Config	Production crawls with many parameters
② Dict API	Quick prototypes, embedded in scripts
③ Static Mode	Pure HTML docs, no browser needed

from generic_crawler import DocCrawler

# ── ① YAML config (recommended) ──
crawler = DocCrawler.from_yaml("my_config.yaml")
crawler.run()

# ── ② Dict directly ──
crawler = DocCrawler.from_dict({
    "base_url": "https://docs.example.com",
    "seed_urls": ["https://docs.example.com/index.html"],
    "url_filters": {"path_prefix": "/docs/"},
    "db_path": "my_docs.db",
})
crawler.run()

# ── ③ Static mode (fast, no Playwright browser needed) ──
crawler = DocCrawler.from_dict({
    "base_url": "https://docs.example.com",
    "seed_urls": ["https://docs.example.com/index.html"],
    "url_filters": {"path_prefix": "/docs/"},
    "db_path": "my_docs.db",
    "render_engine": "requests",
})
crawler.run()

4. Query Results

import sqlite3

conn = sqlite3.connect("my_docs.db")

# Full-text search
results = conn.execute(
    "SELECT title, snippet(pages_search, 2, '<b>', '</b>', '...', 32), url "
    "FROM pages_search WHERE pages_search MATCH ? LIMIT 10",
    ("authentication",)
).fetchall()

# Module breakdown
modules = conn.execute(
    "SELECT module, COUNT(*) FROM pages GROUP BY module ORDER BY 2 DESC"
).fetchall()

---

Rendering Modes

	Playwright Mode	Requests Mode
Best for	JS-rendered docs (MadCap Flare, Docusaurus, VuePress)	Static HTML docs
Speed	~3 pages/sec	~5 pages/sec
Setup	Needs playwright install chromium	No extra setup
Link Discovery	Full rendered DOM (most complete)	Static HTML only

---

Configuration Reference

Parameter	Type	Required	Description
base_url	str	✅	Root URL of target site
seed_urls	list[str]	✅	Entry-point URLs
domain	str		Auto-extracted from base_url
url_filters.path_prefix	str		Only crawl URLs starting with this (e.g. /docs/)
url_filters.allowed_extensions	list		Accepted page extensions
url_filters.skip_patterns	list		Skip URLs containing these substrings
url_filters.skip_path_contains	list		Skip paths containing these keywords
db_path	str		SQLite database path
max_depth	int		Max BFS depth (default 5)
max_pages	int		Max pages to crawl
delay_seconds	float		Delay between pages (default 0.3)
render_engine	str		"playwright" or "requests"
target_language	str		Passed to trafilatura ("en", "zh", etc.)
min_content_length	int		Pages below this get marked low quality (default 150)
content_selectors	list		CSS fallback selectors when trafilatura fails
strip_selectors.tags	list		HTML tags to remove before extraction
strip_selectors.classes	list		CSS classes to remove before extraction

---

Project Structure

generic_crawler/
├── __init__.py          # Package entry, exports DocCrawler
├── crawler.py           # Core BFS crawler class (Playwright + Requests)
├── config.py            # Config system (defaults + YAML loading + validation)
├── example_config.yaml  # Example YAML config
├── requirements.txt     # Python dependencies
├── LICENSE              # MIT
└── README.md            # This file

---

License

MIT — do whatever you want with it.

Inspired by docs-crawler (MIT) architecture pattern.