Three fixes from a crawler bug report against 0.4.1.
session.render(url) loads a page in the browser solver, waits for
client-side rendering to settle, and returns the finished document as a
WaferResponse. Some pages ship a shell and write their content with
JavaScript, so no fingerprint recovers markup that was never in the
bytes: greypointindustries.com went from 20 to 2,877 characters of
visible text, strategicmissions.ca/careers from 539 to 1,879 with its
real job listings, and lodge.tech gained the nav and footer its shell
omits. resp.needs_render flags such a body so a caller knows to escalate.
The render path handles what a browser handles. An interstitial is
solved in place with the same per-WAF handler the transport path uses,
then re-captured, and the session pins its replay identity to the
solving browser so the earned clearance survives the next request
(miata.net: plain TLS gets a 403 challenge, the render returns the real
200 page, the follow-up get() returns 200 with no challenge). Status and
headers always describe the document the body came from, including after
a client-side redirect. A non-HTML resource comes back as the bytes the
server sent under its real content type, since Chrome displays JSON,
text, XML and images inside a generated viewer document.
A Cloudflare WAF block page was classified as generic_js, putting a
terminal denial on the solve-and-rotate path where every attempt bought
another identical 403. ChallengeType.CLOUDFLARE_BLOCK now recognizes it
(error stylesheet present, challenge platform absent, on 403 so a 1015
rate limit keeps its Retry-After path) and RequestBlocked is raised
before any retry, rotation, or session-health accounting. It subclasses
ChallengeDetected, so existing handlers still catch it while a caller
can now tell "try again later" from "this will never work as-is".
ConnectionFailed.reason now names a sinkholed DNS answer. A host that
resolves only to 0.0.0.0 or :: was refused by the resolver, not
unreachable, but the transport can only report a refused connection to
[::]:443, which reads like a dead site. Resolved on the failure path
only, so the happy path pays nothing.