Skip to content

v0.4.0 - pluggable scraper provider with crawl4ai backend

Latest
Latest

Choose a tag to compare

View all tags
@deandevz deandevz released this 06 May 21:37
· 11 commits to main since this release
v0.4.0
951f17f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Adds a pluggable scraper provider abstraction to king-scrape. Firecrawl stays the zero-config default. Crawl4AI is now available as a local opt-in backend (beta) for users who want to index docs without external credits, without sending HTML to a third party, or with full transparency over the HTML to Markdown conversion.

Added

  • Pluggable scraper provider abstraction for king-scrape. Choose backend via SCRAPE_PROVIDER env var or --provider flag. Stage-aware overrides via SCRAPE_DISCOVER_PROVIDER and SCRAPE_FETCH_PROVIDER. Mirrors the layout of src/llm_providers/ (ADR-0009, ADR-0010).
  • Crawl4AI local backend (beta, ADR-0011). Bundled by default in the [all] extra that npx @king-context/cli init runs, so the package ships in the project venv. Activation only requires running crawl4ai-setup once to download the Playwright chromium.
  • .github/workflows/test.yml with a test job (pytest on the firecrawl path) and a smoke-crawl4ai job that runs scripts/smoke-crawl4ai.py to detect API churn across crawl4ai>=0.8.5,<0.9 minor versions.

Changed

  • firecrawl-py moved from core dependencies to the [firecrawl] extra. npx @king-context/cli init continues to install everything via the [all] extra, so the default flow is preserved with no breaking change.
  • installer/lib/python.js now installs king-context[all] via PEP 508 direct reference (king-context[all] @ git+...), so init and update keep pulling firecrawl-py and crawl4ai after the core dependency move.

Enable Crawl4AI

After npx @king-context/cli init (or update from a previous version), run:

.king-context/core/venv/bin/crawl4ai-setup

Then either:

king-scrape https://docs.example.com --provider=crawl4ai

Or set SCRAPE_PROVIDER=crawl4ai in .env to make it the default for the project.

Crawl4AI support is beta. If something breaks or feels off, open an issue at https://github.com/deandevz/king-context/issues with the URL, the command, and the error.

Validation

  • 683 pytest tests passing
  • Crawl4AI end to end on python-httpx.org: 114 sections exported
  • Firecrawl end to end on python-httpx.org: 112 sections exported (zero breaking change confirmed)
  • Public smoke from the npm registry: npx @king-context/cli@0.4.0 init followed by crawl4ai-setup and king-scrape --provider=crawl4ai works end to end on a clean directory

Decisions

  • ADR-0009: Pluggable scraper provider abstraction for king-scrape
  • ADR-0010: Thin scraper provider with pipeline-owned concurrency, checkpoint, and IO
  • ADR-0011: Crawl4AI selected as first local scraping backend

Pull request: #45

Assets 2
Loading