Skip to content

v0.2.0

Latest

Choose a tag to compare

@github-actions github-actions released this 08 Nov 17:18
· 8 commits to main since this release
749e55b

0.2 replaces most of the 0.1 API with fewer types: configuration moves onto SessionBuilder setters, and each job has one name. The upgrade guide maps the changed 0.1.6 names. The sections below list new features, behavior changes, removals without a replacement, and fixes.

Breaking changes

Upgrading from 0.1.6

See UPGRADE.md for the migration steps and complete old-to-new API map.

Added

  • SessionBuilder::run_per_core, which runs one single-threaded runtime per CPU core for maximum request rate; the cores share cookies, cache, tokens and HSTS and keep their own connections.
  • RequestBuilder::text and RequestBuilder::bytes send the request and read the decoded body in one .await.
  • Resumable RequestBuilder::download_to, strong ETag checks, SHA-256 verification, and download progress callbacks. A dropped transfer is retried under the retry policy and resumes when the server sends a strong ETag.
  • An opt-in, bounded HTTP cache with conditional revalidation, Vary matching, and mutation invalidation.
  • RequestBuilder::events, Events, and Event for parsed SSE with Last-Event-ID recovery. The retry policy bounds reconnects in a row; each delivered event resets the count.
  • SessionBuilder::refresh_token shares an asynchronous bearer-token refresh after concurrent 401 responses, scoped to the base URL origin.
  • Crawl::save and Crawl::load preserve pending and buffered pages, duplicate history, crawl policies, and retry waits.
  • HostLimits, Page, Respect, Scope, and SkipReason implement serde Serialize and Deserialize.
  • A Chrome 157 profile, Browser::Chrome157, from Chrome 157.0.8103.38 on Linux and 157.0.8103.39 on macOS and Windows, with TCP and HTTP/3 captures for each desktop.
  • A Brave (Chromium 157) profile, Browser::Brave157, from captures of Brave 1.99.55 (Chromium 157) on macOS, Linux, and Windows.
  • Edge and Opera are Browser variants: Browser::Edge145 to Browser::Edge157, and Browser::Opera129 to Browser::Opera139, on the Chrome captures. Browser::all() lists them.
  • A Firefox 159 profile, Browser::Firefox159, from captures of Firefox 159.0 on macOS, Linux, and Windows.
  • Browser parses from its id with FromStr, as in "chrome-154".parse(). Browser::chromium_major() gives the Chromium base of a Chromium browser, and Browser::next_hello() gives the next distinct ClientHello in the same product.
  • session.save(path) writes the browser, the profile id, the proxy URL without its password, the cookies, and the TLS tickets, Alt-Svc entries, and HSTS store as one JSON file. Session::load(path) returns a SessionBuilder; build() fails with is_profile_changed() when the saved profile id differs, then restores the connection state. session.autosave(path, every) saves on a cookie or state change and on shutdown(). Session implements Serialize and SessionBuilder implements Deserialize, so your own data goes beside the session in your own struct. Version 1 device files load.
  • session.to_builder() returns a SessionBuilder with the session's settings and cookie jar. The new session shares the pool, the TLS session cache, the host limits, and the shutdown state with its parent. A new browser gets its own partition of the pool, and a change to a connection setting gives it a new pool.
  • SessionBuilder::ja3, ja4_r, akamai, and header_order layer a fingerprint on the browser. The session reports the layered profile in profile() and profile_id().
  • Retry::delay(attempt) gives the backoff wait for a retry loop outside a request.
  • Fingerprint::from_json(text) parses an echo service's JSON report, and sent.diff(&observed) lists the fields that differ.
  • RequestBuilder::websocket() opens a WebSocket with the request's headers.
  • Session implements tower::Service with the tower feature.
  • Redirect converts from a usize limit, so .redirect(3) follows up to 3 redirects.
  • A reference page on docs.rs for each core type: Session, SessionBuilder, RequestBuilder, Response, Error, Browser, Platform, Proxy, Tab, Retry, Timeout, Redirect, cookie::Jar, Crawl, and trace::Trace. The crate root is an overview with the job map.
  • AGENTS.md, for coding agents that write code with Leyline.
  • docs/guide/bundled-profiles.md lists every bundled profile and brand overlay, generated from the profile files.
  • Crawl::agent(name) obeys the robots.txt group and meta robots name of a named crawler, and Crawl::max_body_size(n) reads pages larger than the session's cap without a new session.
  • blocking::Session runs requests on the calling thread for code without an async runtime: blocking::Session::new(builder), then session.block_on(future), with every Session method through Deref. Its direct HTTP/1.1 connections read and write blocking sockets, and every timeout still applies. A blocking session is Send but not Sync; build one per thread.
  • leyline-bssl-tokio: RecordStream::try_map_io moves a TLS 1.3 record stream onto another transport for the same connection.
  • Retry::budget limits retries to a share of requests, so retries cannot multiply the load on an overloaded upstream or on the client itself.
  • HostLimits::max_pending and HostLimits::max_total_pending limit the requests that wait for a slot. Without one, a request queue on max_total_in_flight past origin capacity lets every request reach the origin as its deadline ends, and goodput falls to zero.
  • pool_max_lifetime stops reusing a connection a set time after it opened, so traffic moves after a DNS or load balancer change. Without it, a busy pool stays on the old address.
  • max_pending_per_host limits the HTTP/1.1 requests that wait for a connection to one host. A request over the limit fails at once with the new Kind::Overloaded, which gateway_status maps to 503 and retry does not resend. The default is 1024, as in Envoy, so one slow host cannot hold an unbounded queue; max_pending_per_host(None) removes the limit.
  • New connections to a host start on the next address of each family in turn, so a pool spreads its connections across every address DNS returns instead of opening them all on the first.
  • An address that keeps failing to connect moves to the back of the dial order for 30 seconds, longer each time it fails again, so new connections stop waiting on a dead address.
  • TcpProfile has one getter per value: ttl(), mss(), window_size(), df(), window_scale(), no_delay(), and options().

Changed

  • A GET or other idempotent request with a replayable body, on an HTTP/2 connection idle past http2_ping_after_idle, is sent right after its liveness PING and does not wait a round trip for the ACK. A missing ACK fails the connection at http2_ping_timeout, and the request is resent on a new connection. Other requests still wait for the ACK.
  • BoringSSL is pinned to 5b83e5d, the revision in Chromium's DEPS at tag 157.0.8103.38, with six carried patches. The CRL security backport is removed because the fix and its cleanup are included upstream.
  • A browser profile id no longer includes profiles/bare.toml, so an edit to the bare profile does not change browser profile ids. A saved session from before this change reports a changed profile id.
  • The bare profile advertises a 64 KiB HTTP/2 maximum frame size.
  • Body is Send + Sync and is the one body type for requests and responses: bytes_stream() and the tower service both return it.
  • A request uses less CPU: it parses its URL once, keeps its method, URL, and headers in typed values that the response shares, and TLS reads the socket in 16 KiB batches. BENCHMARKS.md compares leyline with other HTTP clients over HTTP/2 and HTTP/1.1.
  • HTTP/2 writes the requests that callers queue together in one write, and while requests wait for a stream slot it reads up to four frames before it writes.
  • A session without a browser profile opens another HTTP/2 connection to a host when every open one is at the server's stream limit, up to max_connections_per_host. Before the server announces its limit, a connection takes 100 requests, as Chrome and Go assume. A browser profile keeps one connection per host unless max_connections_per_host is set, and max_connections_per_host(1) keeps any session on one.
  • After a TLS 1.3 handshake, leyline seals and opens the connection's records itself and BoringSSL handles only the handshake and session tickets. A profile that sets a record size limit keeps BoringSSL's record layer, as does TLS 1.2.
  • Sessions without a browser profile, client certificate, or custom CA files share one TLS context per trust setting. Each session keeps its own TLS session cache.
  • Tracing has one session.execute span per request, and pool.hit is recorded on it. The transport.* and pool.send_request* spans are gone.
  • An HTTP/1.1 or HTTP/2 response header value that is not valid UTF-8 reaches Response::headers() with its bytes unchanged. It was replaced with U+FFFD before.
  • Session::shutdown() returns a future. It still cancels at once; .await it to wait until every autosave writer has made its last save. A call without .await warns.
  • Brave, Firefox, Safari, and Safari on iOS 18 and later race HTTP/3 by default, as Chrome does, from their captured HTTP/3 hellos. Their profile ids change.
  • Every session applies the built-in block rules. A Cloudflare or AWS WAF challenge counts against its proxy in a proxy list, and Attempt::is_block() sees it, without a block_rules call.
  • Error::kind() is the one error classification. Kind gains BodyLimit, Dns, and ProxyTarget, and Kind::gateway_status(). A connect timeout is Kind::Timeout, and is_connect() stays true. Display and Debug of an Error print the kind. The variants are in a new order, so a Kind cast to an integer gives a different value; match on the variant instead.
  • Every way to read a body decodes it: bytes(), text(), bytes_stream(), write_to(writer), and download_to(path). RequestBuilder::decode(false) gives every sink the bytes as sent and keeps the same accept-encoding on the wire.
  • Every sink stops at max_body_size, raw streams included. RequestBuilder::max_body_size(n) replaces the session's value for one request, .stream() responses included. A body over the cap is Kind::BodyLimit, and the message names max_body_size.
  • Each retry waits between half of and all of the computed backoff (equal jitter). A request that never reached the server, one whose TLS handshake timed out included, is retried for any method.
  • Each attempt takes the request's own proxy, else the session's proxy or proxy list. In 0.1, a rotation list replaced a request's own proxy on retries.
  • build() checks the settings, so a bad proxy URL fails there and not at the first send.
  • Each SessionBuilder::trace call adds a hook beside the earlier ones, including a hook that the parent session carries into to_builder().
  • error_for_status() borrows the response, so the body stays readable after a status error.
  • query, form, and Tab::submit use the standard Borrow<(K, V)> bound instead of a custom pair-conversion trait. Pass owned (key, value) tuples or references to them. Explicit generic arguments change from <I, P> to <I, K, V>; inferred calls need no changes.
  • RequestBuilder::header_order takes any iterator of strings, as SessionBuilder::header_order does.
  • Response::text_with_charset needs the charset feature. Without it, the method ignored its argument. text() stays in every build.
  • leyline::get(url) returns a RequestBuilder, so request settings, .text(), and .bytes() chain before .await.
  • multipart::Part::file and Form::file return leyline::Result. A file that cannot be opened is Kind::Io.
  • TcpProfile fields are private; read them through its getters.
  • TlsContext::new(browser, min_version) replaces TlsContext::from_profile, whose profile argument no caller outside the crate could build. A build with only --cfg leyline_unstable_bssl compiles again.

Removed

  • leyline-bssl and leyline-bssl-tokio drop the functions and constants that leyline-http and leyline-quiche never call, about 5,900 lines. Code that calls these crates directly may need the BoringSSL calls from leyline-bssl-sys instead.

Fixed

  • pool_idle_timeout now closes idle connections on a session with no traffic. Before, they stayed open until the next request.
  • A response with several Transfer-Encoding lines is now read as one list, and chunked must end it exactly once. Before, only the first line counted, so chunked then gzip was framed as chunked, a response desync risk.
  • A session that talks to more hosts than max_connections no longer collapses. Eviction scanned every entry under one write lock and closed sockets inside it; at 20,000 hosts and 300,000 req/s on the default limit this held the client to about 190,000 req/s with latency of many seconds. The pool now has up to 64 shards, and sockets close outside the lock.
  • An HTTP/2 connection that uses its last stream id now retires, and later requests move to a new connection. Before, the connection stayed in the pool and every later request to that host failed.
  • retry(n) now reaches the origin at most n + 2 times for an idempotent request: one resend on a stale pooled connection for the first attempt only. Before, every attempt could resend, up to 2 * (n + 1) times.
  • HTTP/2 requests no longer wait until their deadline when more than 1,024 requests queue on a connection and the queue drains in a batch. A sender that waited now wakes the next one, and every waiting sender fails at once when the connection ends.
  • Dropping a session that uses Proxy::identified frees the per-proxy identity sessions and their TLS contexts; before, each derived session kept the pool alive through a reference cycle.
  • A connection failure no longer evicts the healthy HTTP/2 or HTTP/3 connection that replaced the failed one in the pool.
  • A retry that reuses a pooled connection is treated as a pooled attempt: it is replayed after a failure and reported as reused.
  • HTTP/1 rejects malformed status-code widths and version tokens, discards the failed connection, and does not replay a malformed response.
  • The HTTP/2 reader builds with the minimum supported tokio-util version.
  • Reject malformed HTTP/2 response fields and invalid or repeated Content-Length fields instead of accepting incomplete metadata. Apply the shared field validation to HTTP/3 responses and trailers.
  • Reject HTTP/2 connections whose server preface does not start with SETTINGS, and server push after acknowledgement of disabled push.
  • An HTTP/2 server that sends SETTINGS_ENABLE_PUSH with a value other than 0 fails the connection with PROTOCOL_ERROR, as RFC 9113 section 6.5.2 requires.
  • A malformed PRIORITY frame or a zero WINDOW_UPDATE increment on a stream resets only that stream. These errors closed the whole HTTP/2 connection.
  • Concurrent connections to one host and port share a single DNS lookup, including the HTTP/2 and HTTP/3 legs of Protocol::Race.
  • download_to syncs its resume checkpoint at most once a second, writes every chunk that is already received in one blocking call, and hashes a saved prefix while the resume request is in flight. A 256 MiB download over loopback on an M-series Mac went from about 54 MiB/s to about 1,250 MiB/s.
  • download_to finishes a saved prefix of unknown length when the server answers the resume with 416 and Content-Range: bytes */N that equals the prefix. It failed with 416 on every retry.
  • An HTTP/2 RST_STREAM or WINDOW_UPDATE overflow that arrives after END_STREAM no longer discards a streamed body that is still queued for the reader.
  • Benchmark clients use rustls 0.23.46, which fixes certificate decompression, session debug, key-exchange hint, and ECH rejection advisories. Leyline's TLS backend remains BoringSSL.
  • Session::stats() counts one HTTP/2 or HTTP/3 checkout per request. It counted again when a request opened a connection.
  • The Debug output of a response fingerprint printed the authorization and cookie values it sent. Fingerprint prints them as ***.
  • A zstd body whose frame declares a window over 8 MiB fails to decode, the limit RFC 9659 sets for the zstd coding.
  • Cookie Expires dates with an hour over 23, a minute or second over 59, or a year before 1601 are ignored, as RFC 6265 requires.
  • HTTP/2 and HTTP/3 fail a response after 16 informational (1xx) responses, as HTTP/1.1 already did.
  • An HTTP/1.1 chunk-size line follows RFC 9112 section 7.1: a sign, non-hex characters, whitespace before the size, or anything other than spaces, tabs or ; after it is a protocol error.
  • A padded HTTP/2 DATA, HEADERS or PUSH_PROMISE frame too short for its fixed fields is a FRAME_SIZE_ERROR, as RFC 9113 section 4.2 requires; padding longer than the rest of the frame is a PROTOCOL_ERROR.
  • Response::bytes() on a streamed response decodes each chunk as it arrives, so a corrupt compressed body fails at its first bad chunk.