v1.4.0

Added

• A bot wall now says so. When a site's protection refuses the fetch, the
  error names the wall instead of a bare 403 Client Error: Forbidden — e.g.
  "Blocked by Cloudflare — tapology.com uses bot protection and refused this
  request (HTTP 403). Automated access isn't allowed." Cloudflare, DataDome,
  PerimeterX/HUMAN, Akamai and Imperva
  Incapsula are recognised by their response headers and challenge-page markers.
  A new bytecrawl.BlockedError (a subclass of requests.HTTPError) carries it,
  so except requests.HTTPError still catches it and you can catch the block
  specifically. Every surface benefits at once — the library, the crawlers, the
  hosted API and the playground — since the detection lives at the fetch layer.
  Detection is gated on a block-looking status, so a normal page served from
  behind Cloudflare is never mislabelled.
• A rate limit now says so, too. A genuine HTTP 429 Too Many Requests
  (no bot-wall signature) no longer reads as "couldn't reach the site" — it gets
  its own calm message: "The site is rate-limiting us… it's up and reachable,
  just asking for fewer requests. Wait a moment and try again." A new
  bytecrawl.RateLimitError (also a requests.HTTPError subclass) carries the
  server's Retry-After when present — both forms, a delta in seconds and an
  HTTP date. The hosted API answers 429 with a Retry-After header and a
  retry_after field, and the playground shows a wait hint. A 429 that does
  carry a bot-wall signature still reports as blocked — the more specific answer
  wins.
• One error card everywhere. Unreachable seeds (bad DNS, refused
  connection, timeout) now raise a single bytecrawl.UnreachableError across the
  library, crawlers, hosted API and playground, so every playground mode —
  Markdown, links, JSON API, extract, auto and compare — renders the same card
  for the same failure instead of three different-looking errors.
• MCP tools now tell the agent what went wrong. A blocked, rate-limited or
  unreachable seed used to reach an MCP client as a masked "Error executing
  tool …" — the SDK treats a stray requests exception as a crash and withholds
  its text. Every tool (local and hosted) now re-raises these as a ToolError,
  so the agent receives the real message — the wall's name, or the wait hint for
  a 429 — the MCP counterpart of the API's typed 403/429 body.

v1.3.0

Changed

• CI is split into separate checks instead of one "test" job. Lint
  (ruff check), format (ruff format --check), type check (mypy) and
  security (bandit) each run as their own top-level check alongside the
  unit-test matrix (unit test 3.9 / unit test 3.12). A red check now points
  straight at the cause — a formatting slip no longer hides behind a failing
  type check inside a single collapsed job.
• Releases publish to PyPI automatically through GitHub Actions Trusted
  Publishing (OIDC) — no API tokens or stored secrets. A push to main with a
  new __version__ runs the tests, tags the release, and uploads to PyPI;
  nothing publishes without a version bump.

Internal

• Version is single-sourced from __version__: a test now pins the top
  CHANGELOG entry and both READMEs to it, so they can't drift.
• Type-checking fixes across the scraping and crawl modules and one audited
  # nosec on the MCP HTTP bind. No behaviour change.

v1.2.2

Fixed

• fetch_markdown could leak raw JavaScript into its output. On pages
  where the main-content extractor bails (ad-heavy listings, like
  tapology.com's rankings), the fallback converter kept <script> contents as
  loose text instead of dropping them — so the "clean Markdown" could open
  with a wall of ad-tech JS. Fixed; the playground's Markdown preview also
  shows the full result now instead of the first 4,000 characters.

v1.2.1

Wording, and one change to how the agent skill behaves. Nothing in the library
changed — pip install bytecrawl gets the same code as 1.2.0.

Changed

• The agent skill no longer instructs an agent to install it. It opened by
  telling whatever read it to write to ~/.claude/skills/, add MCP servers and
  edit CLAUDE.md "before anything else" — and agents refused, correctly. A
  document fetched from a URL is not a licence to change someone's machine, and
  one agent warned its user that the page was trying to trigger persistent
  changes, which leaves the project looking like the attack rather than the
  tool. Installing is now offered to the person, at the end, and only when they
  ask for it — starting with the hosted MCP server, which writes nothing into
  their home directory and works in every client, with the skill file second
  and the Python package last, since it is only worth installing for JS pages
  or crawls past the hosted cap.
• /bytecrawl invoked bare lists all six options in full instead of folding
  four of them onto one line. "clean text" is not something you can choose
  without guessing what it does. The extraction option now also says you do not
  need to know the selectors.
• The tools section on the landing page is three columns instead of four: there
  are six tools, and four columns left an orphan row of two.

v1.2.0

Added

• Page.selectors() — extraction without having read the markup. Writing an
  extract() call means knowing the classes on the page, and an agent connected
  over MCP could not: fetch_markdown strips exactly the classes and attributes
  a selector is built from, and no tool returns HTML on purpose, since raw markup
  costs more tokens than the data it is meant to locate. So this reports the
  structure instead of the source — the blocks that repeat, the inner selectors
  that resolve across most instances, and a sample record for each, which is a
  runnable item + fields rather than something to interpret.

  Two heuristics do the work. A repeated wrapper and the card inside it appear
  equally often, so the one that offers nothing its own child does not is
  dropped: article.product_pod survives, li.col-xs-6.col-sm-4.col-md-3 does
  not. And a column whose element contains another column's is thrown away,
  because its text is every child's text run together — div.product_price
  otherwise comes back as "£51.77In stockAdd to basket".

Fixed

• markdown() was throwing the page away on listings. Main-content
  extraction is built to discard repetition — which on a listing page is the
  content. On books.toscrape.com, the first URL the README tells you to curl,
  it returned the price column and dropped all twenty book titles: 358
  characters out of 1,851 of visible text. It now checks what came back against
  the page's own text and falls back to full-page conversion when the
  extractor kept under a quarter of it. Measured, that floor sits far below
  every good case (1.14x of visible text on quotes.toscrape.com, 1.4-1.6x on
  Wikipedia) and just above the one that broke (0.19x), so an article buried in
  navigation still gets the boilerplate stripped.

  With the content back, the saving is 4.8x on books.toscrape.com, 5.8x on
  quotes and 6.6x on a Wikipedia article — the 5-10x the README claims. The
  broken version scored 144x, which was not a saving.

Fixed

• "Setup for agents" set nothing up. The button copies Read and follow <url>, which makes an agent read the skill for exactly one turn — next
  message it has none of it, and /bytecrawl never appears. The skill showed
  the install command but never told the agent to run it, so the button promised
  setup and delivered a single answer. Installing itself is now the first
  instruction in the file — and it branches on what the agent is, because a
  skills directory is Claude Code's convention and telling Codex or Cursor to
  write ~/.claude/skills/ produces a file nobody reads. Claude Code installs
  the file and gets /bytecrawl; an MCP client with no skills directory adds
  the hosted server, which is the one path that persists everywhere and hands
  over the tools rather than instructions about them; an agent with an
  AGENTS.md adds one line pointing back; a sandbox with nothing to write to
  skips it silently. If a task came with the URL, the task comes first.

• /docs advertised v1.0.0 through three releases. The version was written
  into the template, so nothing tied it to the package and it simply stopped
  being true. It renders from bytecrawl.__version__ now, and a test fails if
  a literal version reappears there.

Changed

• fetch_markdown returns tokens_html next to tokens_estimate, the pair the
  playground already showed. On its own the token count has nothing to compare
  against; beside the raw HTML it is the reason to have called the tool, and an
  agent can state the saving instead of asserting it.
• extract called with only a url now discovers instead of failing. It used
  to raise "extract needs 'item' and 'fields', or 'select'" — refusing the one
  question an agent can ask about a page it has not seen. The same call works on
  every surface: the MCP tool, /api?method=extract without select, and the
  playground's Extract chip with the field left empty, which lists the blocks it
  found and turns each into a button that fills the selector and runs it.
• The demo API no longer sorts JSON keys. The discovered fields come back ranked
  with the most useful selector first and the playground offers that one as the
  button; sorted alphabetically, books.toscrape.com led with a::attr(href).

v1.1.2

Fixed

• The agent skill sent agents to a repo that 404s. SKILL.md — the file
  handed to an agent as the entry point — pointed at a GitHub URL that does not
  exist, so anything that went looking for the source or an issue tracker hit a
  dead end.
• It documented API response fields that no method returns. The skill
  promised status and elapsed; every response actually carries url and
  method, which were undocumented, and compare was not documented at all.
  The troubleshooting steps then told you to read page.status to diagnose a
  failed call — so the first debugging move the skill taught did not work over
  HTTP.

Added

• A smoke suite against the deployed site (pytest -m smoke, excluded from CI
  like the browser tests). Every bug in this release and the last was invisible
  to the offline suite: they lived in the gap between what the project says and
  what the deploy does. These check that the skill's links resolve, that the
  API returns the fields it documents, that the hosted MCP server lists its six
  tools, and that the SSRF guard is live.

v1.1.1

Fixed

• The hosted MCP endpoint answered every request with 421 Invalid Host header. The transport enables DNS-rebinding protection whenever the app is
  built for its default 127.0.0.1 host, and then allows only localhost — so
  nothing reaching bytecrawl.vercel.app got through. That protection is for a
  server bound to your own machine; this one is public, unauthenticated and
  already behind an SSRF guard, so it now ships open. Set
  BYTECRAWL_ALLOWED_HOSTS (and BYTECRAWL_ALLOWED_ORIGINS) to turn it back on
  if you self-host on a domain that also serves authenticated apps.

v1.1.0

MCP: six tools instead of four. Point any MCP-capable agent at ByteCrawl and
it can now do everything the playground does. Nothing to install:

claude mcp add --transport http bytecrawl https://bytecrawl.vercel.app/mcp

Tool	What it does
focused_crawl	Crawl a site, rank pages by relevance to a query
compare_strategies	New. All three strategies, one budget, side by side
fetch_markdown	One page → clean Markdown
extract	Structured records, or a flat list from one selector
list_links	New. Every outbound link, absolute and deduplicated
fetch_json_api	Hit a hidden JSON API

Local (pip install bytecrawl[mcp], then claude mcp add bytecrawl -- bytecrawl-mcp)
exposes the identical six, with no page cap and browser support.

Added

• compare(url, query) — run Shark-Search, OPIC and BFS over one site on the
  same budget and see which actually finds the topic. On Wikipedia + "san
  francisco" that is 6 relevant pages out of 6 for Shark-Search against 1 of 6
  for BFS. The three run concurrently, so it costs about as long as one crawl.
  Returns a winner, or tied when nothing separates them.
• extract accepts a single select selector for a flat list of values, as an
  alternative to item + fields.
• HTTP API: /api?url=SITE&method=compare&query=TOPIC for the same comparison
  without any install.

Changed

• Page.links() now returns absolute, deduplicated URLs and drops
  #fragments, mailto: and asset files — the same normalisation the crawlers
  already used. On a long Wikipedia article that is 2,222 raw hrefs down to
  1,408 pages you can actually visit. Pass raw=True for the old output.
• normalize() moved to bytecrawl.core (still importable from
  bytecrawl.crawler), so links and crawling share one definition of a link.

Fixed

• The demo site: a jumping footer, a webfont that resized the logo on first
  paint, and the strategy comparison now looks the same on every page.

v0.3.0

Changed

• Project renamed: ByteCraw → ByteCrawl ("craw" read like a typo; "crawl"
  matches what the library does and the naming of the space it lives in).
  New identity everywhere: package bytecrawl on PyPI, repo
  github.com/abrahamperz/ByteCrawl (the old URL redirects), demo at
  bytecrawl.vercel.app, MCP command bytecrawl-mcp. The old bytecraw
  0.1.1 package on PyPI is unmaintained — install bytecrawl.

v0.2.1

Changed

• Docs only: README and changelog now state explicitly that the MCP server is
  local (stdio transport) — no hosted/remote endpoint ships yet.