v0.1.11

Added

• SOURCELIB_BROWSER=headed forces the solver to open a window up front, for a run someone is watching.

Fixed

• render reached only a stage's first page. The pages after it came from a plain GET, which a challenged host refuses.
• A field template naming a sibling declared below it is refused at load time. It rendered empty, and where the field was required every row dropped.

v0.1.10

Fixed

• A form was always sent as UTF-8, whatever encoding said. RFC-0001 section 3.6 requires the opposite — values in form are encoded with the source's encoding before percent-encoding — and a site serving a legacy encoding expects its search terms in it. Nothing failed: measured against a GBK host, the search endpoint answered 1,336 bytes and no rows for a UTF-8 body and 3,954 bytes with the novel for the same query as GBK, so the stage returned an empty result rather than an error. A query the named encoding cannot spell, or a codec that does not exist, falls back to the old behaviour rather than failing a stage the caller can still use. The request body changes, so a recorded search on a source that declares a non-UTF-8 encoding has to be re-recorded.

v0.1.9

Changed

• The fetch extra requires scraper 1.8, where on_decoy: "warn" reports a suspected decoy without refusing it and a verdict expires. Under 1.7 a page the guard flagged once was refused on every later run, which took whole hosts out of reach: a chapter whose prose shares little vocabulary with the table of contents it was found on reads as off-topic, and three measured at 13,403, 11,210 and 13,190 characters were refused on the run after they were read successfully.

v0.1.8

Added

• count, a list step giving how many entries arrived. For the pager that reports its length by how many options it has rather than by what any of them says: several themes render one <option> per page and give every one the same value, leaving the page total in the count and nowhere else. Such a stage had to fall back to while, which walks speculatively; on one host the same novel took over ten minutes that way and twenty-one seconds with last.

v0.1.7

Added

• The browser solver runs in auto mode: hidden first, a window only when that fails and only where a display exists, so a challenge that wants a person can be cleared by one.
• ItemList.script, naming an element whose JSON carries the rows, for a page that renders its list from a payload rather than into markup. RFC-0001 section 3.8.

Fixed

• A stage's default request no longer renders the address it was given. A URL read off a page went through the template renderer a second time, so a site whose markup contains braces had its own text read as a placeholder and the crawl failed naming a placeholder the spec never wrote.
• A solver is configured only when its transport is installed too. Without it the solver was built and raised on first use, so a host the browser never reached looked like a host that refused.

v0.1.6

Added

• try-search, running a spec's search stage against a live query. Takes a path and a query, no novel URL.
• A browser solver is configured when the machine has one, so request.render works and a challenged host can be reached. Needs lncrawl-scraper[cdp] and a Chromium-family build; absent, nothing changes.

Fixed

• json with css on one item list parses, then selects. It selected first and found nothing.
• An extraction no longer edits the document. A node pipe rewrote the shared tree, so a stage reusing it through page read a page an earlier stage had rewritten: one host's chapter list went from 77 rows to none, and the toc was blamed.
• const is interpolated, as section 3.4 has always said, so a field can be produced from vars alone.
• Templates are validated at load time. An unknown placeholder, an unknown filter and a placeholder used outside its stage all passed check and were found by a crawl.

Changed

• A var's const may not name another var, and is refused at load time. RFC-0001 section 3.5.

v0.1.5

Added

• paginate.step, how much {page} advances between pages.
• require on an item list, naming further fields a row cannot do without.
• A strip_matching step, removing an element whose text matches a pattern.
• check_response and login are called.

Fixed

• A pipe naming a step that does not exist passed validation.
• A require naming an undeclared field is refused at load time.
• The JSON Schema rejected the null that deletes an inherited key.
• A pipe fed by all: true was type-checked as though it took a scalar.

Changed

• spec increments on a change in meaning, not on an addition.

v0.1.4

Changed

• paginate.count is renamed last, and paginate.limit is gone.

Added

• A min transform step, the counterpart to max.
• paginate.first, the number a site gives its first page.
• Pages are fetched in parallel wherever the termination condition allows. paginate.concurrent: false turns it off.
• try --toc-pages N caps the chapter-list walk.

Fixed

• rate_limit was never passed to the fetcher.
• --sample capped at three. Samples now spread evenly across the list.
• drop_leading could delete a whole chapter and still report ok.

v0.1.3

Changed

• lxml is the default parser, and a dependency. A fixture recorded before this release will not replay.

Fixed

• Five hook points were legal, bindable and never called.
• A parsed fragment carried the parser's <html><body> wrapper into its output.
• paragraphs leaked a BeautifulSoup warning to stderr.

v0.1.2

Fixed

• from gave up on the first alternative that failed unexpectedly.
• from never checked whether an alternative produced items.
• A URL template's doubled slash reached the site.
• A page naming a request that had not run failed mid-crawl rather than at load time.