Skip to content

Octocrawl 0.4.0

Choose a tag to compare

View all tags
@77777R7 77777R7 released this 10 Oct 08:19
· 937 commits to main since this release
v0.4.0
c3e1383
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Octocrawl 0.4.0, released 2026-10-10: readable Markdown and checked fields from public web pages for AI agents, RAG and research, with an Evidence Record on every page.

This is the release the roadmap's enhanced-access phase names once its first items have their real-site records:

  • sessions and egress (item 3);
  • the one access choice standard / enhanced / my-browser (item 7);
  • reading a page in your own Chrome with my-browser (item 8).

Their code shipped in 0.3.1 and 0.3.2. The records are in research/access/runs/ and research/parity/runs/.

What changed since 0.3.2 fixes pages that answered success without the data asked for:

  • Pages that only name a refusal or a browser check are judged a block on every lane. These are short 2xx pages led by a heading such as "Pardon Our Interruption", "Robot or human?" or eBay's "Checking your browser", which had answered success.
  • Products a page declares in its JSON-LD are read as a listing. This covers products under a CollectionPage or SearchResultsPage, and a top-level ItemList of three or more. The product grid becomes the page's content.
  • A main region the page hides (hidden on it or around it) is no longer taken for the page's content.
  • Two more kinds of script-filled page go on to the browser rung instead of answering success with headers and no figures: one whose content still says it is loading (WSJ's market data), and a thin page whose only state is a bare window.__data = (the World Bank's indicator pages).
  • A provider session that cannot be opened is the provider's failure (failed/provider_error), not an HTTP 500.

Packages: npm octocrawl, @octocrawl/cli, @octocrawl/sdk and @octocrawl/mcp; PyPI octocrawl-client.

npx octocrawl scrape https://example.com --markdown
claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp
pip install 'octocrawl-client[pandas]'

@octocrawl/mcp 0.4.0 is listed in the official MCP Registry as io.github.77777R7/octocrawl.

Every change since 0.3.2 is listed in CHANGELOG.md.

Website and docs: https://octocrawl.dev

Assets 2
Loading