Skip to content

Releases

v0.2.0

Choose a tag to compare

@github-actions github-actions released this 08 Nov 17:18
749e55b

0.2 replaces most of the 0.1 API with fewer types: configuration moves onto SessionBuilder setters, and each job has one name. The upgrade guide maps the changed 0.1.6 names. The sections below list new features, behavior changes, removals without a replacement, and fixes.

Breaking changes

Upgrading from 0.1.6

See UPGRADE.md for the migration steps and complete old-to-new API map.

Added

  • SessionBuilder::run_per_core, which runs one single-threaded runtime per CPU core for maximum request rate; the cores share cookies, cache, tokens and HSTS and keep their own connections.
  • RequestBuilder::text and RequestBuilder::bytes send the request and read the decoded body in one .await.
  • Resumable RequestBuilder::download_to, strong ETag checks, SHA-256 verification, and download progress callbacks. A dropped transfer is retried under the retry policy and resumes when the server sends a strong ETag.
  • An opt-in, bounded HTTP cache with conditional revalidation, Vary matching, and mutation invalidation.
  • RequestBuilder::events, Events, and Event for parsed SSE with Last-Event-ID recovery. The retry policy bounds reconnects in a row; each delivered event resets the count.
  • SessionBuilder::refresh_token shares an asynchronous bearer-token refresh after concurrent 401 responses, scoped to the base URL origin.
  • Crawl::save and Crawl::load preserve pending and buffered pages, duplicate history, crawl policies, and retry waits.
  • HostLimits, Page, Respect, Scope, and SkipReason implement serde Serialize and Deserialize.
  • A Chrome 157 profile, Browser::Chrome157, from Chrome 157.0.8103.38 on Linux and 157.0.8103.39 on macOS and Windows, with TCP and HTTP/3 captures for each desktop.
  • A Brave (Chromium 157) profile, Browser::Brave157, from captures of Brave 1.99.55 (Chromium 157) on macOS, Linux, and Windows.
  • Edge and Opera are Browser variants: Browser::Edge145 to Browser::Edge157, and Browser::Opera129 to Browser::Opera139, on the Chrome captures. Browser::all() lists them.
  • A Firefox 159 profile, Browser::Firefox159, from captures of Firefox 159.0 on macOS, Linux, and Windows.
  • Browser parses from its id with FromStr, as in "chrome-154".parse(). Browser::chromium_major() gives the Chromium base of a Chromium browser, and Browser::next_hello() gives the next distinct ClientHello in the same product.
  • session.save(path) writes the browser, the profile id, the proxy URL without its password, the cookies, and the TLS tickets, Alt-Svc entries, and HSTS store as one JSON file. Session::load(path) returns a SessionBuilder; build() fails with is_profile_changed() when the saved profile id differs, then restores the connection state. session.autosave(path, every) saves on a cookie or state change and on shutdown(). Session implements Serialize and SessionBuilder implements Deserialize, so your own data goes beside the session in your own struct. Version 1 device files load.
  • session.to_builder() returns a SessionBuilder with the session's settings and cookie jar. The new session shares the pool, the TLS session cache, the host limits, and the shutdown state with its parent. A new browser gets its own partition of the pool, and a change to a connection setting gives it a new pool.
  • SessionBuilder::ja3, ja4_r, akamai, and header_order layer a fingerprint on the browser. The session reports the layered profile in profile() and profile_id().
  • Retry::delay(attempt) gives the backoff wait for a retry loop outside a request.
  • Fingerprint::from_json(text) parses an echo service's JSON report, and sent.diff(&observed) lists the fields that differ.
  • RequestBuilder::websocket() opens a WebSocket with the request's headers.
  • Session implements tower::Service with the tower feature.
  • Redirect converts from a usize limit, so .redirect(3) follows up to 3 redirects.
  • A reference page on docs.rs for each core type: Session, SessionBuilder, RequestBuilder, Response, Error, Browser, Platform, Proxy, Tab, Retry, Timeout, Redirect, cookie::Jar, Crawl, and trace::Trace. The crate root is an overview with the job map.
  • AGENTS.md, for coding agents that write code with Leyline.
  • docs/guide/bundled-profiles.md lists every bundled profile and brand overlay, generated from the profile files.
  • Crawl::agent(name) obeys the robots.txt group and meta robots name of a named crawler, and Crawl::max_body_size(n) reads pages larger than the session's cap without a new session.
  • blocking::Session runs requests on the calling thread for code without an async runtime: blocking::Session::new(builder), then session.block_on(future), with every Session method through Deref. Its direct HTTP/1.1 connections read and write blocking sockets, and every timeout still applies. A blocking session is Send but not Sync; build one per thread.
  • leyline-bssl-tokio: RecordStream::try_map_io moves a TLS 1.3 record stream onto another transport for the same connection.
  • Retry::budget limits retries to a share of requests, so retries cannot multiply the load on an overloaded upstream or on the client itself.
  • HostLimits::max_pending and HostLimits::max_total_pending limit the requests that wait for a slot. Without one, a request queue on max_total_in_flight past origin capacity lets every request reach the origin as its deadline ends, and goodput falls to zero.
  • pool_max_lifetime stops reusing a connection a set time after it opened, so traffic moves after a DNS or load balancer change. Without it, a busy pool stays on the old address.
  • max_pending_per_host limits the HTTP/1.1 requests that wait for a connection to one host. A request over the limit fails at once with the new Kind::Overloaded, which gateway_status maps to 503 and retry does not resend. The default is 1024, as in Envoy, so one slow host cannot hold an unbounded queue; max_pending_per_host(None) removes the limit.
  • New connections to a host start on the next address of each family in turn, so a pool spreads its connections across every address DNS returns instead of opening them all on the first.
  • An address that keeps failing to connect moves to the back of the dial order for 30 seconds, longer each time it fails again, so new connections stop waiting on a dead address.
  • TcpProfile has one getter per value: ttl(), mss(), window_size(), df(), window_scale(), no_delay(), and options().

Changed

  • A GET or other idempotent request with a replayable body, on an HTTP/2 connection idle past http2_ping_after_idle, is sent right after its liveness PING and does not wait a round trip for the ACK. A missing ACK fails the connection at http2_ping_timeout, and the request is resent on a new connection. Other requests still wait for the ACK.
  • BoringSSL is pinned to 5b83e5d, the revision in Chromium's DEPS at tag 157.0.8103.38, with six carried patches. The CRL security backport is removed because the fix and its cleanup are included upstream.
  • A browser profile id no longer includes profiles/bare.toml, so an edit to the bare profile does not change browser profile ids. A saved session from before this change reports a changed profile id.
  • The bare profile advertises a 64 KiB HTTP/2 maximum frame size.
  • Body is Send + Sync and is the one body type for requests and responses: bytes_stream() and the tower service both return it.
  • A request uses less CPU: it parses its URL once, keeps its method, URL, and headers in typed values that the response shares, and TLS reads the socket in 16 KiB batches. BENCHMARKS.md compares leyline with other HTTP clients over HTTP/2 and HTTP/1.1.
  • HTTP/2 writes the requests that callers queue together in one write, and while requests wait for a stream slot it reads up to four frames before it writes.
  • A session without a browser profile opens another HTTP/2 connection to a host when every open one is at the server's stream limit, up to max_connections_per_host. Before the server announces its limit, a connection takes 100 requests, as Chrome and Go assume. A browser profile keeps one connection per host unless max_connections_per_host is set, and max_connections_per_host(1) keeps any session on one.
  • After a TLS 1.3 handshake, leyline seals and opens the connection's records itself and BoringSSL handles only the handshake and session tickets. A profile that sets a record size limit keeps BoringSSL's record layer, as does TLS 1.2.
  • Sessions without a browser profile, client certificate, or custom CA files share one TLS context per trust setting. Each session keeps its own TLS session cache.
  • Tracing has one session.execute span per request, and pool.hit is recorded on it. The transport.* and pool.send_request* spans are gone.
  • An HTTP/1.1 or HTTP/2 response header value that is not valid UTF-8 reaches Response::headers() with its bytes unchanged. It was replaced with U+FFFD before.
  • Session::shutdown() returns a future. It still cancels at once; .await it to wait until every autosave writer has made its last save. A call without .await warns.
  • Brave, Firefox, Safari, and Safari on iOS 18 and later race HTTP/3 by default, as Chrome does, from their captured HTTP/3 hellos. Their profile ids change.
  • Every session applies the built-in block rules. A Cloudflare or AWS WAF challenge counts against its proxy in a proxy list, and Attempt::is_block() sees it, without a block_rules call.
  • Error::kind() is the one error classification. Kind gains BodyLimit, Dns, and ProxyTarget, and Kind::gateway_status(). A connect timeout is Kind::Timeout, and is_connect() stays true. Display and Debug of an Error print the kind. The variants are in a new order, so a Kind cast to an integer gives a different va...
Read more

v0.1.6

Choose a tag to compare

@github-actions github-actions released this 25 Oct 14:22
41390d3

Added

  • A Brave (Chromium 156) profile, Browser::Brave156, from captures of Brave
    1.98.60 (Chromium 156) on macOS, Linux, and Windows.
  • A Chrome 156 profile, Browser::Chrome156, from captures of Chrome
    156.0.8078.48 on macOS, Linux, and Windows.
  • ChromiumBrand::Opera covers Chrome 154 and sends Opera 138 there.

Changed

  • Browser::default() and Browser::latest(Family::Chrome) send Chrome 156, and
    Browser::latest(Family::Brave) sends Brave 156.
  • BoringSSL is pinned to 2ea3200, the boringssl_revision in Chromium's DEPS
    at tag 156.0.8078.47, and carries 6 patches.

Fixed

  • An HTTP/2 connection whose peer stopped reading kept taking new requests
    when the request it stalled was small enough for the socket buffers to
    hold, so each later request on it waited for its own timeout. A request
    that times out before its response headers arrive now makes the pool send
    a PING before it reuses that connection, and the pool opens a new
    connection when the PING gets no answer within h2_ping_timeout.

v0.1.5

Choose a tag to compare

@github-actions github-actions released this 13 Oct 22:39
f85af55

Added

  • Session::crawl returns Crawl, a polite stream of CrawlPages that obeys
    robots.txt and each page's robots rules by default, with five settings:
    ignore_robots() turns them off; respect(Respect) adds AI-training
    opt-outs (AiTraining); scope(Scope); limit(pages); and
    host_limits(HostLimits). By default it sends one request at a time and one
    a second per host where the session's own HostLimits leave it unset,
    pauses a host on 429 or 503 and tries the page up to three times, sends
    every request through both the session's own HostLimits and the crawl's
    pace, keeps the site's Crawl-delay out of the session's own requests,
    waits out a robots.txt outage before it skips a site, reads sitemaps,
    follows redirects as links, and drops error pages, files, block pages, and
    copies by canonical URL or text, preferring the canonical page over an alias
    found first. CrawlStats counts each SkipReason.
  • Session::robots and Robots read robots.txt by RFC 9309: group matching,
    with every group record (including Crawl-delay, Content-Signal, and
    Content-Usage) closing the run of User-agent lines, longest match with
    * and $, percent-encoding, the 4xx and 5xx rules, up to five redirects,
    and a 512 KiB limit. Robots also reads Content-Signal and the IETF draft
    Content-Usage as Purpose, Crawl-delay, and Sitemap lines;
    Robots::allows_use takes the URL.
  • html::text returns the visible text of a page, one block per line, with
    <pre> kept as written and script, style, and head text left out. An
    unclosed <head> ends at the first element that cannot be in it.
  • html::Directives reads noindex, nofollow, none, and noai from
    <meta name="robots">, a meta tag named for the agent, and
    X-Robots-Tag. html::canonical returns the page's canonical link.
  • A Firefox 158 profile, Browser::Firefox158, from captures of Firefox 158.0
    on macOS, Linux, and Windows.

Changed

  • html::links, html::forms, and other readers built on the HTML scanner
    now keep text after a comment or a bare < (<a>a < b</a> gives a < b),
    skip </ followed by a non-letter to the next >, and keep a namespace
    prefix in tag names (fb:like), as the HTML standard does.

  • A HostLimits::host override now applies to IPv6 literal hosts; before, it
    never matched them.

  • leyline-bssl-sys compiles BoringSSL with the cc crate from the source
    lists BoringSSL ships, so a build needs only a C and C++ compiler and Git:
    no CMake. On Windows, NASM is optional; without it the build links the
    NASM objects that ship in the crate. The CMake-only variables
    CMAKE_TOOLCHAIN_FILE, CMAKE_GENERATOR, and
    LEYLINE_BSSL_COMPILER_EXTERNAL_TOOLCHAIN no longer apply; use CC,
    CXX, CFLAGS, and CXXFLAGS. The build rebuilds BoringSSL when its
    source, the compiler, or the compiler flags change, and reuses the cached
    library otherwise, so a restored build cache never links a stale BoringSSL.

v0.1.4

Choose a tag to compare

@github-actions github-actions released this 09 Oct 03:39
8334e04

Added

  • A Brave (Chromium 155) profile, Browser::Brave155, from captures of Brave
    1.97.56 (Chromium 155) on macOS, Linux, and Windows.
  • A Chrome 155 profile, Browser::Chrome155, from captures of Chrome
    155.0.8059.40 on macOS, Linux, and Windows.
  • ChromiumBrand::Opera covers Chrome 153 and sends Opera 137 there.

Changed

  • Browser::default() and Browser::latest(Family::Chrome) send Chrome 155, and
    Browser::latest(Family::Brave) sends Brave 155.
  • BoringSSL is pinned to b780f19, the boringssl_revision in Chromium's DEPS
    at tag 155.0.8059.39, and carries 6 patches.

v0.1.3

Choose a tag to compare

@github-actions github-actions released this 07 Oct 13:53
b926648

Security

  • A saved TLS session restored into a session with different trust
    settings could resume there, and a resumed handshake skips the
    certificate check, so pins and custom roots did not apply to it. Each
    saved TLS session is now bound to the trust its session loaded: the
    pins, the contents of every root from add_ca_der, add_ca_file,
    SSL_CERT_FILE, and SSL_CERT_DIR, and the contents of the client
    certificate and key, hashed from the same bytes the TLS context loaded.
    It resumes only in a session that loaded the same trust, and
    Session::fresh_pool keeps the trust of the session it copies. Saved
    state contains TLS session secrets; store it with the same protection
    as credentials. A saved ticket for a proxied connection no longer holds
    the proxy user name; it holds a hash of the proxy credentials, as the
    accounts guide states.

Added

  • RequestBuilder::read_until(limit, done) streams the response and returns
    a PrefixRead: the decoded prefix and a StopReason (PredicateMatched,
    LimitReached, or EndOfBody). It keeps the request's redirects, retries,
    authentication, host limits, and error_for_status().
    Response::read_until keeps its signature, and now calls done only
    after a chunk that adds decoded bytes, as RequestBuilder::read_until
    does.
  • examples/stock_monitor.rs.

Fixed

  • A .stream() body that failed on a timeout, a decode error, or shutdown
    kept its connection and, for a decoded body, its host-limit slot until
    the caller dropped the stream. The failed stream now releases both at
    once, and the trace reports the real outcome instead of Dropped.
  • WebSocketBuilder::connect and Session::preconnect ignored
    Session::shutdown(). Both now fail with the shut-down error after
    shutdown, including a preconnect that would do nothing, and a WebSocket
    handshake takes a host-limit slot while it runs. An established
    WebSocket connection stays open.
  • An HTTP/2 connection whose peer stopped reading blocked the whole
    connection inside a socket write: no response, reset, ping, or shutdown
    got through. The connection now keeps reading and cancelling while
    output waits, stops taking new work and reading new frames while its
    write buffer is full, and parks an upload instead of queuing it past
    that budget. After a second without write progress it takes no new
    requests, so the pool opens a fresh connection and queued requests are
    refused unsent and retried there; it closes once no stream needs it.
  • An HTTP/3 connection waiting for stream credit moved every queued
    request into an unbounded queue, and each failed stream started a task
    that could wait forever on a full body channel. Queued requests now stay
    in the bounded request channel, and a failed stream reports its error
    through a reserved slot.
  • An HTTP/1.1 upload did not read the response until the whole request
    body was sent, so a server that answered 413 early and stopped reading
    left the request hanging until its timeout. Leyline now reads the
    response while it uploads, stops the upload at a final response or when
    the server closes its side, and closes that connection. A body that
    yields many chunks without pausing no longer keeps the request from
    seeing the response, its timeout, or shutdown.
  • A streamed HTTP/2 or HTTP/3 upload whose body yielded empty chunks
    without pausing kept its upload task running after the request was
    cancelled. The upload task now yields on an empty chunk and stops when
    the request is gone.
  • A cookie attribute value over 1024 octets is now ignored, as RFC 6265bis
    requires, and the jar refuses cookies past the size limits from every
    source, including extend_from and saved jars.
  • An HTTP/2 response that ended with trailers skipped the
    Content-Length check, so a short body arrived as a success. It now
    fails like a short body without trailers.
  • A Body::stream with a declared length that produced more or fewer
    bytes sent a malformed request over HTTP/2 and HTTP/3. Every protocol
    now fails the request with a body error and sends nothing past the
    declared length.
  • Jar::autosave and Device::autosave lost the last change when the
    runtime stopped before the autosave task first ran. The final save now
    runs in that case too.
  • TestServer::next_request returned a request only after the
    response's delay, just before the reply went out. It now returns the
    request when it arrives, as documented.
  • error_for_status over HTTP/1.1 waited for a small error body with no
    error-body timeout, so a stalled 500 with a short Content-Length ran to
    the total timeout and returned Kind::Timeout without the status. The
    error-body window now bounds that read, and the error keeps the status and
    the bytes received.
  • error_for_status over HTTP/1.1 read error bodies up to 64 KiB regardless
    of max_error_body. It now reads at most max_error_body bytes with the
    response, and never more than max_body_size. An error body that Leyline
    does not decode, with compression turned off or an unknown coding, keeps
    its Content-Encoding header.
  • A status error whose body read timed out or failed dropped the bytes
    already decoded. It now keeps them.

Changed

  • An HTTP/2 request whose header list size (RFC 7541 §4.1: name, value,
    and 32 octets per field) is larger than the server's
    SETTINGS_MAX_HEADER_LIST_SIZE, or than 256 KiB when the server sets
    none, now fails before any of it is encoded or sent. Buffered
    HTTP/2 uploads use DATA frames no larger than the free space in the
    connection's 64 KiB output budget, even when the server allows larger
    frames.
  • The timeout guide now states that read and body apply to a
    .stream() response and to a body the transport delivers in pieces; a
    body the transport buffers is bounded by response_header and total.
  • Response::download_to commits on a blocking thread after the body is
    written and synced: a future dropped before the commit starts leaves the
    target unchanged, and a commit that has started finishes. A
    directory-sync failure after the rename says that the target was
    already replaced.
  • Over HTTP/1.1, a .stream() body dropped before its end closes the
    connection. Leyline drains the rest only for error_for_status error
    bodies, up to max_error_body bytes.

v0.1.2

Choose a tag to compare

@github-actions github-actions released this 05 Oct 18:26

Security

  • leyline-bssl-sys carries BoringSSL's fix for CVE-2026-35189. A peer
    certificate with many nameRelativeToCRLIssuer CRL distribution points
    could make certificate verification allocate memory without bound. CRL
    distribution points that use a relative name are now ignored.

Added

  • A Firefox 157 profile, Browser::Firefox157, from captures of Firefox 157.0
    on macOS and Linux.

Changed

  • leyline-bssl-sys ships pre-generated BoringSSL bindings for each
    supported target, so a build no longer compiles or runs bindgen, and
    links the BoringSSL source into the build directory instead of copying
    it. Cold builds are faster. To generate the bindings at build time, for
    example for another target, enable the new bindgen feature. CMake uses
    the Ninja generator when ninja is on PATH and CMAKE_GENERATOR is
    not set.

Fixed

  • leyline-http builds again with default features off and http3 left
    out. 0.1.1 did not compile in that configuration.
  • RequestBuilder::send boxes its request state, so nesting a few requests
    in one task no longer overflows a 2 MB thread stack.
  • RetryPolicy::retry_unsent(true) no longer replays a non-idempotent
    request after a redirect leg got a response. A connection failure on a
    later leg used to resend the original request, for example a POST that
    the server had already processed and answered with a 303. A 401 digest
    challenge does not count as an answer, so a request whose authorized leg
    could not connect is still retried.
  • error_for_status reads and decodes at most max_error_body bytes of an
    error response, within the error-body timeout, on HTTP/1.1, HTTP/2, and
    HTTP/3. It used to read and decode the whole body first, so a large, slow,
    or highly compressed error body ran to the body timeout or returned a body
    error instead of the status error. A response below 400 is still read in
    full, with its trailers and connection reuse. The error's headers no longer
    list a content-encoding that its decoded body does not have.
  • Each Browser variant keeps a fixed discriminant, listed in
    profiles/browser_ids.toml, so adding a profile no longer changes the
    value of the variants after it.
  • The leyline-bssl-sys build script reruns when a patch is added to
    patches/. Before, a new patch was not applied to an existing build
    directory.

v0.1.1

Choose a tag to compare

@github-actions github-actions released this 03 Oct 00:09
ac6bd69

Changed

  • Session::new() and Session::default() build a plain session.
    Session::browser(browser) is the one-line browser session:
    Session::browser(Browser::default()) is the newest bundled Chrome on
    Windows, which Session::new() built before. A browser takes the first
    platform its own profile covers, so a mobile browser gets its mobile
    platform, with or without the builder.
  • A plain session never sends browser-only headers (sec-*, client hints,
    upgrade-insecure-requests, priority), for any preset.
  • On a browser session, SessionBuilder::user_agent drops the sec-ch-ua
    client hint, so the user agent and the hint never disagree.
  • Firefox accept-language weights follow Firefox: equal steps from 1, as in
    en,ja;q=0.5 and de-DE,de;q=0.7,en;q=0.3.
  • A session with a browser and no protocol call takes its protocol policy
    from the profile: it races HTTP/3 when the http3 feature is on and the
    profile's [h3] table sets race = true.
  • Sessions derived with with_identity share the parent's connection pool,
    HSTS store, and Alt-Svc knowledge, in a partition per profile, instead of
    opening a new pool.
  • Converting an io::Error that wraps a leyline::Error returns the wrapped
    error with its own kind instead of Kind::Io.

Added

Sessions and requests

  • leyline::get(url) sends one GET from a plain session.
  • SessionBuilder::base_url, bearer_auth, user_agent, and languages.
    With a base_url, the session token goes only to the base URL's origin. Session::with_base_url derives a session with another base.
  • RequestBuilder::error_for_status() turns a 4xx or 5xx into an error
    after the retries, and keeps the headers and the start of the body.
    CompressionConfig::max_error_body caps the kept body (64 KiB by default)
    for error_for_status and download, and TimeoutConfig::error_body
    bounds its read (10 s by default, and never past the total timeout).
    Decompression stops at the cap. The kept body is cut at the smaller of
    max_error_body and max_body_size, with or without content-encoding.
  • RequestBuilder::download(path, limit) and Response::download_to write
    the decoded body through a temporary file, so the path holds the whole
    body or nothing; a declared Content-Length over the cap is refused before
    reading. Response::into_decoded_stream and copy_decoded_to stream the
    decoded body under a limit.
  • RequestBuilder::pages() follows Link: rel="next", and counts the first
    URL as fetched, so a redirect does not repeat the first page.
    Pages::limit(max) stops after max pages; there is no default limit;
    Response::link(rel) and links() parse Link headers.
  • RequestBuilder::cookie_jar, initiator, and tag. Response::attempts()
    and Response::proxy().
  • TimeoutConfig::body(duration) bounds the time to read a whole body,
    counted from the first read of the body.
  • Session::shutdown() stops every clone of a session, including bodies
    being streamed; Error::is_shut_down() names that error.
  • relay_headers(headers, RelayBody) and Response::relay_headers return the
    headers to forward from a proxy or a service. proxy-authenticate and
    proxy-authorization are dropped as hop-by-hop. RelayBody is
    #[non_exhaustive]. With RelayBody::Decoded,
    Response::relay_headers keeps content-encoding when the session did not
    decode the body.
  • RequestBuilder::download and Response::download_to keep the permission
    bits of a target file that already exists, never setuid, setgid, or sticky
    bits.
  • multipart::Part::file(path) reads a file part that takes a MIME type.

Errors and retries

  • Error::category() returns one ErrorCategory per failure, with
    as_str() and gateway_status(). Error::is_connect() agrees with it. Error::is_dns(), is_proxy(), and
    is_body_limit() refine it. Error::find() reaches a leyline error inside
    any error chain, such as a tower BoxError. leyline::Error converts from
    url::ParseError.
  • A status error keeps headers(), header(name), body(), body_text(),
    attempts(), and proxy(), and carries the retry policy's view of its last
    response: retry_after() and retries_exhausted().
  • RetryPolicy::retry_if(predicate), wait_header(name, WaitFormat),
    retry_unsent(true), skip_blocks(rules), backoff(attempt), and
    rotate_proxies(list). A wait_header wins over Retry-After.
    retry_unsent(true) retries any method after an error the server did not
    process: DNS, connect, TLS, proxy, a connect timeout, an HTTP/2
    REFUSED_STREAM reset, or an HTTP/3 request the server reports as not
    processed. An HTTP/3 request the server rejects twice is retried on the
    same connection.
  • RetryPolicy::transient() caps a server-requested wait at 60 s
    (max_retry_after). Error::retries_exhausted() is true when the policy
    wanted another try and did not make it: no retries left, a server wait above
    max_retry_after, or a wait or backoff longer than the time left.
  • A streaming request body that was never read, for example after a connect
    error, is retried.

Proxies, crawling, and blocks

  • SessionBuilder::host_limits(HostLimits): requests in flight and per second
    per origin, overrides per host, a total cap, and pause_on(statuses) to
    stop new requests to an origin for the wait the server asks for, read from
    the retry policy's wait headers or Retry-After. pause_for sets the pause
    when the server asks for none (60 s), and max_pause caps a requested
    pause (24 h). A streamed request holds its slot until its body ends; a
    streamed body ends at its first error and releases the slot. A failed
    attempt releases its slot before the retry wait.
    Session::host_stats() reports requests in flight and waiting per origin.
  • Host limits admit each redirect hop against its own origin. pause_on and
    proxy-pool strikes apply to the origin that sent the response. A request
    waits for its origin slot, then for any pause and its next rate slot, then
    the total slot. No request holds a total slot while it waits for a pause or the rate spacing. The spacing is
    measured at admission, and a pause is checked again just before admission.
    A request that loses its turn while it waits for the total slot waits
    again. A host name with a trailing dot
    is the same host. Each session built from a HostLimits value has its own
    counters, shared by its clones and derived sessions.
  • ProxyPool: sticky proxies, bans after repeated failures, rotation on block
    rules, and ProxyPool::identified to pin a browser identity to each proxy,
    each proxy with its own cookie jar, also when two proxies share an
    identity. ProxyPool::identified on a session without a browser fails at
    build() with Kind::Config. ProxyPool::stats() reports health.
    Identity sessions share one connection pool, partitioned by profile.
  • Response::block() reports a bot-protection challenge from vendors'
    response headers. BlockRules::from_toml, statuses([..]), extend, and
    check define and apply your own rules; one BlockRules value drives
    skip_blocks, rotate_on_block, and check.
  • PoolStats::busy and PoolStats::idle.

Browser sessions, tabs, and devices

  • HeaderAnchor::BeforeCchUa places a header before sec-ch-ua, where
    Chrome sends the client hints a site asked for.
  • Session::tab() returns a Tab that keeps the current page: open,
    follow, submit, submit_form, fetch, xhr, post_json, and
    subresource send the page as the initiator. A tab with no page refuses
    script requests.
  • Jar::save_to, load_from, and autosave keep a jar on disk.
    Jar and device files are versioned, { "version": 1, .. }, and are written
    with mode 0600 on Unix (default ACLs on Windows). autosave(path, interval) saves at most once per interval after a change, and saves
    again when the runtime shuts down.
  • Device holds a browser device (identity, frozen profile, profile_id,
    user agent, proxy and proxy_password_env, languages, jar, browser state,
    current page, and caller data in app) and reopens it as the same session.
    Device::check refuses a session that differs, and strict refuses a
    device without a pinned profile or a proxy. Device::autosave keeps the
    device and its jar on disk. Serializing a Device with
    proxy_password_env set never writes the proxy password; DeviceAutosave::update and track change it.
    It takes a Duration or DeviceAutosaveOptions, whose state_interval
    (300 s by default) saves the connection state on a period.
    SessionBuilder::expect_profile_id and Error::is_profile_changed() catch
    a changed profile.
  • Session::state() returns SessionState: TLS session tickets, HTTP/3
    Alt-Svc knowledge, and the HSTS store. Sessions keep an HSTS store, and
    learn no HSTS entry when certificate verification is off. Saved TLS session
    keys for an authenticated proxy hold the proxy user name, never the
    password or a value derived from it.
  • Browser, Family, Platform, ChromiumBrand, Identity, ProxyUrl,
    and leyline::Url implement serde; the profile enums have stable string
    ids. SessionIdentity::to_identity() and profile_id().
  • The html feature (on by default, and it enables multipart):
    html::forms, Form::find, html::meta, and html::links. Forms follow
    the browser rules: form="id" controls (the first element with that id),
    disabled fieldsets (only the first <legend> that is a direct child is
    exempt), disabled <optgroup> options left out, a list-box <select size>
    with no selected option left out, one checked radio button per name, line
    breaks in names and values converted to CRLF, and every HTML named
    character reference.
    Form::set replaces every value of a name. Tab::submit_form encodes the
    fields as the form's `FormEnc...
Read more