Skip to content

v0.1.1

Choose a tag to compare

@github-actions github-actions released this 03 Oct 00:09
· 151 commits to main since this release
ac6bd69

Changed

  • Session::new() and Session::default() build a plain session.
    Session::browser(browser) is the one-line browser session:
    Session::browser(Browser::default()) is the newest bundled Chrome on
    Windows, which Session::new() built before. A browser takes the first
    platform its own profile covers, so a mobile browser gets its mobile
    platform, with or without the builder.
  • A plain session never sends browser-only headers (sec-*, client hints,
    upgrade-insecure-requests, priority), for any preset.
  • On a browser session, SessionBuilder::user_agent drops the sec-ch-ua
    client hint, so the user agent and the hint never disagree.
  • Firefox accept-language weights follow Firefox: equal steps from 1, as in
    en,ja;q=0.5 and de-DE,de;q=0.7,en;q=0.3.
  • A session with a browser and no protocol call takes its protocol policy
    from the profile: it races HTTP/3 when the http3 feature is on and the
    profile's [h3] table sets race = true.
  • Sessions derived with with_identity share the parent's connection pool,
    HSTS store, and Alt-Svc knowledge, in a partition per profile, instead of
    opening a new pool.
  • Converting an io::Error that wraps a leyline::Error returns the wrapped
    error with its own kind instead of Kind::Io.

Added

Sessions and requests

  • leyline::get(url) sends one GET from a plain session.
  • SessionBuilder::base_url, bearer_auth, user_agent, and languages.
    With a base_url, the session token goes only to the base URL's origin. Session::with_base_url derives a session with another base.
  • RequestBuilder::error_for_status() turns a 4xx or 5xx into an error
    after the retries, and keeps the headers and the start of the body.
    CompressionConfig::max_error_body caps the kept body (64 KiB by default)
    for error_for_status and download, and TimeoutConfig::error_body
    bounds its read (10 s by default, and never past the total timeout).
    Decompression stops at the cap. The kept body is cut at the smaller of
    max_error_body and max_body_size, with or without content-encoding.
  • RequestBuilder::download(path, limit) and Response::download_to write
    the decoded body through a temporary file, so the path holds the whole
    body or nothing; a declared Content-Length over the cap is refused before
    reading. Response::into_decoded_stream and copy_decoded_to stream the
    decoded body under a limit.
  • RequestBuilder::pages() follows Link: rel="next", and counts the first
    URL as fetched, so a redirect does not repeat the first page.
    Pages::limit(max) stops after max pages; there is no default limit;
    Response::link(rel) and links() parse Link headers.
  • RequestBuilder::cookie_jar, initiator, and tag. Response::attempts()
    and Response::proxy().
  • TimeoutConfig::body(duration) bounds the time to read a whole body,
    counted from the first read of the body.
  • Session::shutdown() stops every clone of a session, including bodies
    being streamed; Error::is_shut_down() names that error.
  • relay_headers(headers, RelayBody) and Response::relay_headers return the
    headers to forward from a proxy or a service. proxy-authenticate and
    proxy-authorization are dropped as hop-by-hop. RelayBody is
    #[non_exhaustive]. With RelayBody::Decoded,
    Response::relay_headers keeps content-encoding when the session did not
    decode the body.
  • RequestBuilder::download and Response::download_to keep the permission
    bits of a target file that already exists, never setuid, setgid, or sticky
    bits.
  • multipart::Part::file(path) reads a file part that takes a MIME type.

Errors and retries

  • Error::category() returns one ErrorCategory per failure, with
    as_str() and gateway_status(). Error::is_connect() agrees with it. Error::is_dns(), is_proxy(), and
    is_body_limit() refine it. Error::find() reaches a leyline error inside
    any error chain, such as a tower BoxError. leyline::Error converts from
    url::ParseError.
  • A status error keeps headers(), header(name), body(), body_text(),
    attempts(), and proxy(), and carries the retry policy's view of its last
    response: retry_after() and retries_exhausted().
  • RetryPolicy::retry_if(predicate), wait_header(name, WaitFormat),
    retry_unsent(true), skip_blocks(rules), backoff(attempt), and
    rotate_proxies(list). A wait_header wins over Retry-After.
    retry_unsent(true) retries any method after an error the server did not
    process: DNS, connect, TLS, proxy, a connect timeout, an HTTP/2
    REFUSED_STREAM reset, or an HTTP/3 request the server reports as not
    processed. An HTTP/3 request the server rejects twice is retried on the
    same connection.
  • RetryPolicy::transient() caps a server-requested wait at 60 s
    (max_retry_after). Error::retries_exhausted() is true when the policy
    wanted another try and did not make it: no retries left, a server wait above
    max_retry_after, or a wait or backoff longer than the time left.
  • A streaming request body that was never read, for example after a connect
    error, is retried.

Proxies, crawling, and blocks

  • SessionBuilder::host_limits(HostLimits): requests in flight and per second
    per origin, overrides per host, a total cap, and pause_on(statuses) to
    stop new requests to an origin for the wait the server asks for, read from
    the retry policy's wait headers or Retry-After. pause_for sets the pause
    when the server asks for none (60 s), and max_pause caps a requested
    pause (24 h). A streamed request holds its slot until its body ends; a
    streamed body ends at its first error and releases the slot. A failed
    attempt releases its slot before the retry wait.
    Session::host_stats() reports requests in flight and waiting per origin.
  • Host limits admit each redirect hop against its own origin. pause_on and
    proxy-pool strikes apply to the origin that sent the response. A request
    waits for its origin slot, then for any pause and its next rate slot, then
    the total slot. No request holds a total slot while it waits for a pause or the rate spacing. The spacing is
    measured at admission, and a pause is checked again just before admission.
    A request that loses its turn while it waits for the total slot waits
    again. A host name with a trailing dot
    is the same host. Each session built from a HostLimits value has its own
    counters, shared by its clones and derived sessions.
  • ProxyPool: sticky proxies, bans after repeated failures, rotation on block
    rules, and ProxyPool::identified to pin a browser identity to each proxy,
    each proxy with its own cookie jar, also when two proxies share an
    identity. ProxyPool::identified on a session without a browser fails at
    build() with Kind::Config. ProxyPool::stats() reports health.
    Identity sessions share one connection pool, partitioned by profile.
  • Response::block() reports a bot-protection challenge from vendors'
    response headers. BlockRules::from_toml, statuses([..]), extend, and
    check define and apply your own rules; one BlockRules value drives
    skip_blocks, rotate_on_block, and check.
  • PoolStats::busy and PoolStats::idle.

Browser sessions, tabs, and devices

  • HeaderAnchor::BeforeCchUa places a header before sec-ch-ua, where
    Chrome sends the client hints a site asked for.
  • Session::tab() returns a Tab that keeps the current page: open,
    follow, submit, submit_form, fetch, xhr, post_json, and
    subresource send the page as the initiator. A tab with no page refuses
    script requests.
  • Jar::save_to, load_from, and autosave keep a jar on disk.
    Jar and device files are versioned, { "version": 1, .. }, and are written
    with mode 0600 on Unix (default ACLs on Windows). autosave(path, interval) saves at most once per interval after a change, and saves
    again when the runtime shuts down.
  • Device holds a browser device (identity, frozen profile, profile_id,
    user agent, proxy and proxy_password_env, languages, jar, browser state,
    current page, and caller data in app) and reopens it as the same session.
    Device::check refuses a session that differs, and strict refuses a
    device without a pinned profile or a proxy. Device::autosave keeps the
    device and its jar on disk. Serializing a Device with
    proxy_password_env set never writes the proxy password; DeviceAutosave::update and track change it.
    It takes a Duration or DeviceAutosaveOptions, whose state_interval
    (300 s by default) saves the connection state on a period.
    SessionBuilder::expect_profile_id and Error::is_profile_changed() catch
    a changed profile.
  • Session::state() returns SessionState: TLS session tickets, HTTP/3
    Alt-Svc knowledge, and the HSTS store. Sessions keep an HSTS store, and
    learn no HSTS entry when certificate verification is off. Saved TLS session
    keys for an authenticated proxy hold the proxy user name, never the
    password or a value derived from it.
  • Browser, Family, Platform, ChromiumBrand, Identity, ProxyUrl,
    and leyline::Url implement serde; the profile enums have stable string
    ids. SessionIdentity::to_identity() and profile_id().
  • The html feature (on by default, and it enables multipart):
    html::forms, Form::find, html::meta, and html::links. Forms follow
    the browser rules: form="id" controls (the first element with that id),
    disabled fieldsets (only the first <legend> that is a direct child is
    exempt), disabled <optgroup> options left out, a list-box <select size>
    with no selected option left out, one checked radio button per name, line
    breaks in names and values converted to CRLF, and every HTML named
    character reference.
    Form::set replaces every value of a name. Tab::submit_form encodes the
    fields as the form's FormEnctype says (URL-encoded, multipart/form-data,
    or text/plain) and resolves the action against <base href>.

Tracing, audit, and testing

  • Trace::summary receives one trace::Summary per request, and
    Trace::body one trace::BodyEnd per streamed body. trace::Metrics
    counts requests, statuses, error categories, attempts, body outcomes, and
    latency; trace::Fanout sends events to several traces.
  • AuditData::compare(&audit::Observed) compares a session's JA4, JA3,
    HTTP/2 fingerprint, request headers, and header order
    (FingerprintReport::header_order) with an echo service report. A header
    the service did not echo is FieldOutcome::Absent, which counts as a
    mismatch.
  • The test-util feature adds leyline::testing::TestServer, a local HTTP or
    HTTPS server with a private CA that records requests, with delayed and
    chunked responses. TestServer::http_on(listener, handler) serves on your
    own listener. RecordedRequest has raw (the request head as received),
    request_line, header_values, header_count, and text;
    TestResponse::close closes the connection after the response.
    leyline::redact_url exposes the crate's URL redaction.

Fixed

  • leyline-http requires tokio-util 0.7.5, the first version with the
    owned cancellation future it uses; 0.7.0 to 0.7.4 failed to build.
  • A request that follows a URL the server chose (a pages() next link,
    Tab::follow, or Tab::submit_form) drops the session credentials when
    its origin differs from the page it came from, as a redirect does: the
    Authorization, Cookie, and Proxy-Authorization defaults and the
    session bearer token.
  • On HTTP/2 and HTTP/3, a Host header you set is not sent; the request
    authority carries the host.
  • HeaderStyle::Bare is the header shape of a plain session. HeaderStyle
    variants have fixed discriminants.
  • A browser session's connection to an origin that speaks only HTTP/1.1 sends
    the profile's own ClientHello (its ALPN list and ALPS) and carries the
    request on that connection, as a browser does. Before, Leyline dialed a
    second connection that offered only http/1.1, a ClientHello no browser
    sends, and a streamed request body to such an origin failed.
  • A plain session sends no accept-language unless you set languages or the
    header. A socks5:// proxy set on the builder or in a ProxyPool without
    the socks feature fails at build().
  • A request with an initiator page computes sec-fetch-site and Origin
    from that page even when the referrer policy sends no Referer, as when an
    https page requests an http URL. A navigation with an initiator is a link
    navigation: it sends the computed sec-fetch-site and a Referer in the
    position each browser sends it. A navigation without one is a typed URL, as
    before.
  • A header template value whose placeholders all render empty is not sent. A
    plain session no longer sends empty or Chromium-only client hints when a
    preset is set.
  • A plain session's accept-encoding lists only the codings that are compiled
    in and enabled in its CompressionConfig, and is omitted when none are.
  • HTTP/3: a queued request whose caller has gone is not sent, and responses
    report their connection timing and reuse.
  • HTTP/2 and HTTP/3 do not send a caller's host header next to the
    :authority.
  • ProtocolPolicy::Race skips HTTP/3 for 5 minutes for an origin and proxy
    pair after an HTTP/3 connection to it fails. The Http3 policy error names
    the missing socks feature when that is the cause.
  • A proxy that reports it could not reach the origin (SOCKS5 replies 3 to 6,
    CONNECT 502 or 504) gives TlsError::ProxyTargetUnreachable with a
    ProxyReply; Error::is_proxy() is false for it. A timeout while reaching
    a proxy on the TLS path is a proxy error and still a timeout.
  • A body-limit error names the limit that applied: the session's
    max_body_size or the caller's limit.
  • Dropping an HTTP/1.1 streamed response releases its connection and the
    per-host connection slot at once, even while the server sends nothing.
  • LeylineService streams the decoded body, reads a ProxyConfig from the
    request extensions, and puts the final Url, HttpVersion, and
    ResponseTiming in the response extensions. Session::execute reads the
    same request extensions.
  • Saving a jar skips expired cookies, and loading one skips expired cookies,
    keeps the newest of two cookies in one slot, and applies the cookie limits.
    Each cookie's last-access time survives a save, so eviction order survives
    a restart. A jar that 0.1.0 serialized with serde still deserializes.
  • HTTP/2: a request that the caller cancels while it waits for a stream slot
    is not sent. Before, the driver sent it when a slot became free.
  • HTTP/2: a streaming request body buffers at most 256 KiB ahead of the
    peer's flow-control window, the same limit HTTP/3 uses. Before, a stalled
    peer let the driver buffer the whole body. The body stream is first polled
    when the request starts on the wire, not when it is queued.
  • HTTP/2: when the caller drops a request whose upload has not finished, the
    driver resets the stream with CANCEL and stops polling the body.
  • HTTP/2: when every stream slot is in use, at most one request waits inside
    the driver. Later requests wait in the bounded command queue, and the
    caller's timeout applies to that wait. Pings use their own queue, so a pool
    liveness ping does not wait behind queued requests.
  • The pool remembers that an origin negotiated HTTP/1.1 for 10 minutes, then
    offers HTTP/2 again. It keeps at most 1024 such origins.

Build

  • The minimum tokio is 1.37 and the minimum enum_dispatch for
    leyline-quiche is 0.3.13. Leyline did not compile against the older
    versions the 0.1.0 manifests allowed.