Octocrawl 0.4.1

Octocrawl 0.4.1, released 2026-10-10: a fix release on top of 0.4.0.

• No more connection resets while polling a crawl. A client of the local API or the local MCP host could have its next request on a kept-alive connection reset (TypeError: fetch failed, read ECONNRESET). This happened about one run in two when a crawl's status was polled after reading its pages mid-run. The cause was the server's own work holding its event loop past the keep-alive timeout. An idle connection is now closed only when no request has arrived on it. Hosted Octocrawl has had this fix since 2026-10-10.
• Products listed at the top level of a page's JSON-LD (three names or more, one script each, as on Redfin's search results) are read as a listing's cards.
• The page's <main> wins over chrome around it. It is preferred when every h1 of the page is in it, as on Sephora's category pages.

Packages: npm octocrawl, @octocrawl/cli, @octocrawl/sdk and @octocrawl/mcp; PyPI octocrawl-client.

npx octocrawl@latest scrape https://example.com --markdown
pip install --upgrade 'octocrawl-client[pandas]'

@octocrawl/mcp 0.4.1 is listed in the official MCP Registry as io.github.77777R7/octocrawl, and https://mcp.octocrawl.dev/mcp serves 0.4.1.

Every change since 0.4.0 is listed in CHANGELOG.md.

Website and docs: https://octocrawl.dev

Octocrawl 0.4.0

Octocrawl 0.4.0, released 2026-10-10: readable Markdown and checked fields from public web pages for AI agents, RAG and research, with an Evidence Record on every page.

This is the release the roadmap's enhanced-access phase names once its first items have their real-site records:

• sessions and egress (item 3);
• the one access choice standard / enhanced / my-browser (item 7);
• reading a page in your own Chrome with my-browser (item 8).

Their code shipped in 0.3.1 and 0.3.2. The records are in research/access/runs/ and research/parity/runs/.

What changed since 0.3.2 fixes pages that answered success without the data asked for:

• Pages that only name a refusal or a browser check are judged a block on every lane. These are short 2xx pages led by a heading such as "Pardon Our Interruption", "Robot or human?" or eBay's "Checking your browser", which had answered success.
• Products a page declares in its JSON-LD are read as a listing. This covers products under a CollectionPage or SearchResultsPage, and a top-level ItemList of three or more. The product grid becomes the page's content.
• A main region the page hides (hidden on it or around it) is no longer taken for the page's content.
• Two more kinds of script-filled page go on to the browser rung instead of answering success with headers and no figures: one whose content still says it is loading (WSJ's market data), and a thin page whose only state is a bare window.__data = (the World Bank's indicator pages).
• A provider session that cannot be opened is the provider's failure (failed/provider_error), not an HTTP 500.

Packages: npm octocrawl, @octocrawl/cli, @octocrawl/sdk and @octocrawl/mcp; PyPI octocrawl-client.

npx octocrawl scrape https://example.com --markdown
claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp
pip install 'octocrawl-client[pandas]'

@octocrawl/mcp 0.4.0 is listed in the official MCP Registry as io.github.77777R7/octocrawl.

Every change since 0.3.2 is listed in CHANGELOG.md.

Website and docs: https://octocrawl.dev

Octocrawl 0.3.1

Octocrawl 0.3.1, released 2026-10-06: readable Markdown and checked fields from public web pages for AI agents, RAG and research, with an Evidence Record on every page.

Packages: npm octocrawl, @octocrawl/cli, @octocrawl/sdk and @octocrawl/mcp; PyPI octocrawl-client.

npx octocrawl scrape https://example.com --markdown
claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp
pip install 'octocrawl-client[pandas]'

@octocrawl/mcp 0.3.1 carries mcpName: io.github.77777R7/octocrawl. Since 2026-10-09 it is listed in the official MCP Registry under that name, with the stdio package and the hosted server at https://mcp.octocrawl.dev/mcp.

Every change since 0.3.0 is listed in CHANGELOG.md. That list is too long for a release page.

Website and docs: https://octocrawl.dev

Octocrawl Gate 2–4 MVP v0.4.0-rc.1 (source pre-release, published as W2L)

This source prerelease freezes the Gate 2–4 execution, delivery, and onboarding MVP.

Highlights:

• Explicit capture modes and end-to-end cancellation/deadline propagation.
• Persistent Retry-After handling, real process recovery, claim fencing, controlled A/B/A/B changes, HTTP 304 cached-body verification, and Monitor isolation.
• Durable HTTPS webhook delivery with leases, retries, dead-letter replay, stable event IDs, receiver deduplication, and restart recovery.
• Crawl/Monitor/Delivery SDK flows, runnable examples, onboarding documentation, and a sanitized agent clean-install record.

Validation:

• GitHub Local Reliability Gate passed twice for PR #50.
• Local npm run typecheck passed.
• Local npm test passed: 79 files / 925 tests.
• The 346-file acceptance snapshot, archive, manifest, and test-log hashes were verified.

Boundaries:

• Workspace packages remain private and are not published to npm by this release.
• Independent non-author Gate 4 acceptance remains pending.
• Gate 5 external two-week/repeat-use validation has not started.
• Monitor/Delivery MCP tools, unified process management, remote HTTPS URL MCP, and persistent public hosting are planned C2/C3 work and are not included.

See docs/roadmap/gate-2-4-acceptance.md and docs/roadmap/section-c-delivery.md in this tag for the evidence and next slice.

Publication-status documentation was subsequently corrected on main by PR #51 (03eb826) so the live roadmap records this merge and prerelease. The tagged runtime source is unchanged.