Skip to content

Releases: jose-pr/syshttp

Release list

v0.1.0

Choose a tag to compare

@github-actions github-actions released this 20 Oct 13:48

Added

  • syshttp.client: an HTTP client on the standard library, so that no feature
    of syshttp needs requests (which only syshttp.autopatch patches, in a
    program that imports it). HTTPClient(transport=None, *, headers, timeout, deadline, user_agent, env, proxymap, ssl_context, proxy_ssl_context, verify, cookies, tokens, auth_provider, retries, backoff, status_forcelist, cookie_hosts) sends through a transport and follows redirects with one rule
    for credentials (Authorization, Cookie and Proxy-Authorization go to the
    origin they were given for and to no other scheme, host or port). The default HTTPTransport (on http.client,
    with a per-thread connection pool, framed bodies, gzip / deflate
    decoding and a bounded reply size) routes every request and redirect hop by
    proxy.proxy_map(env=env) (or proxymap=): the first entry that accepts a
    connection, an http or https proxy (CONNECT; an https proxy over TLS
    with the origin's TLS inside it), a URL-prefix gateway, a proxy's
    credential to the proxy alone; and verifies with the caller's ssl_context,
    else the CA_BUNDLE bundle or verify=<path>, else the system's context
    (proxy_ssl_context for an https proxy or gateway). Cookies (RFC 6265,
    per origin, on every hop, in a jar by host with cookies=), tokens, _PY
    hooks, auth_provider and retries (Retry-After honoured, a status that
    outlasts them returned) work as they did on the session. HTTPResponse
    reads its body whole as content, text() and json().
  • syshttp.fetch() (defined in syshttp.client, re-exported from the root
    without importing it): fetch(url, method="GET", *, headers, data, json, timeout, client, transport, reauth, **client_options) returns the
    HTTPResponse with its body read, whatever the status; a given client is
    left open and a new one is closed; reauth runs once after a 401 or 403.
  • The transport protocol is public: Transport (send() and close()),
    PreparedRequest and prepare_request() give another transport (on
    requests, httpx, or a fake) the checks and body framing of syshttp's own.
    SYSHTTP_DEFAULT_TRANSPORT (module:attr or <file>.py:attr, read under the
    prefixes) names the transport of a client given none; a value that cannot be
    used is ConfigurationError. hooks.load_object() is the loader it shares
    with the _PY hooks.
  • syshttp.exceptions: SysHTTPError (the base of what goes wrong at run
    time), ConfigurationError (a ValueError), HookError (a RuntimeError)
    and ResponseError (a ValueError), exported from the root. A malformed
    setting, a hook that cannot be loaded, a failing hook and a body that is not
    JSON raise them; every one is still the ValueError or RuntimeError it
    was, so existing handlers keep working. A caller's own mistake stays a plain
    ValueError / TypeError.
  • Every public module declares __all__ (proxy also lists its setting-name
    constants, PROXY and DEFAULT_AUTH_HEADER among them).
  • syshttp.env.HTTPEnv(prefixes=None, environ=None): every setting is a
    variable with a full name -- syshttp's own carry SYSHTTP_
    (SYSHTTP_PROXY, SYSHTTP_HOME, SYSHTTP_HOOKS_<KEY>,
    SYSHTTP_IMPORTPATCH), conventional ones keep theirs (HTTPS_PROXY,
    NO_PROXY, HTTP_PROXY_AUTH, CA_BUNDLE) -- read as <PREFIX><NAME> for
    each prefix of SYSHTTP_PREFIX in turn, a :-separated list used
    literally (default: one empty prefix, the names as written). The first
    prefix that sets a variable wins, each variable on its own:
    SYSHTTP_PREFIX=MYTOOL_ reads only MYTOOL_<NAME>, MYTOOL_: falls back
    to the unprefixed names. find() returns the variable found with its
    value, for messages. Every function that reads a setting takes env=,
    new_session() and the fetch *_auto helpers included.
  • CA_BUNDLE (under the prefixes) names the CA bundle a session's
    verify=True uses; sessions no longer read requests' own
    REQUESTS_CA_BUNDLE / CURL_CA_BUNDLE.
  • The system's proxy configuration is the last proxy source, decided for the
    URL being requested: WinHTTP on Windows, CFNetwork on macOS, libproxy or the
    desktop's settings elsewhere, PAC scripts and WPAD included (through
    proxylib, a new dependency). It comes after SYSHTTP_PROXY and the
    *_PROXY variables, after the NO_PROXY check, and is always spoken to as
    connect. SYSHTTP_SYSTEM_PROXY=0 turns it off. proxy.plan(),
    proxy_url(), resolve() and proxy_authorization() take the request URL
    (url= / target=). proxy.configured_proxy_url() gives the
    environment's answer alone. The pac extra adds a JavaScript engine for PAC
    scripts proxylib evaluates itself (Linux without libproxy).
  • syshttp.autopatch: SYSHTTP_IMPORTPATCH=1 (or :-separated names:
    requests, urllib, httpx, aiohttp; read under the prefixes like
    every setting) patches those libraries as they are imported,
    in any program of the environment, so requests without a proxy or TLS
    setting of the caller's route by syshttp and verify with the system's
    context. Through importpatch, a new dependency, and its entry points;
    patch() / unpatch() do it from code.
  • The system's answer is a list (PROXY a; PROXY b; DIRECT): the first proxy
    that accepts a connection is used (proxylib's first_working_proxy), and a
    DIRECT reached first is direct. No reachable proxy and no DIRECT raises
    proxy.NoReachableProxyError (a ConnectionError, exported from the
    root); a session raises requests.exceptions.ProxyError for it.
  • new_session(ssl_context=..., proxy_ssl_context=...) (and the same on the
    fetch *_auto helpers): a caller's ssl.SSLContext is used as given for
    every https origin, or for an https:// proxy. That covers mutual TLS,
    ciphers and protocol versions. It cannot be combined with a verify other
    than True.
  • hooks.wrapper_environment(): the SYSHTTP_HOOK_* variables a _CURL
    wrapper run receives.
  • jar.default_root(): the per-user state directory used when
    SYSHTTP_HOME is unset.
  • syshttp.__version__.
  • Hosts that share cookies: SYSHTTP_COOKIE_HOSTS (hosts split by ,,
    groups by ;) or new_session(cookie_hosts=...) declares names that reach
    one service. In a group, a cookie a member sets for itself or another
    member belongs to the first host, every member's requests carry the
    group's cookies, and any member can delete them; a Domain outside the
    group is still judged on the host that sent it. cookies.host_aliases()
    builds the mapping, which set_cookie_specs, merge_set_cookie_headers
    and host_only_policy take as aliases=.
  • The shipped header and the settings page list every variable syshttp
    reads under its own name (the proxy family, REQUEST_METHOD, the CA bundle
    variables, the state-directory variables, importpatch's and proxylib's).

Changed

  • syshttp.fetch is a function, not a module: fetch() replaces the helpers
    below. _PY hooks are called as hook(client, method, url, kwargs) with the
    keyword arguments of HTTPClient.request(), and may return an
    HTTPResponse; auth.AuthProvider.ensure(client, url) takes the client.
    CA_BUNDLE is the bundle a client's verify=True uses. cookies.py keeps
    the Netscape rows, set_cookie_specs, cookie_applies and host_aliases
    only; the client keeps its cookies as those rows.

  • Routing is proxylib 2's (proxylib>=2.0.2,<3; 2.0.1 breaks pip's cached
    requests). proxy.proxy_map(proxy, noproxy, proxy_user, *, env) is the settings as one proxylib proxy map,
    read at each lookup; proxy.configured_proxy() is SYSHTTP_PROXY as a
    proxylib Proxy; proxy.system_map() is the system's configuration
    (proxylib.auto_proxy). The *_PROXY variables are read per scheme
    (lowercase first, httpoxy-safe), NO_PROXY takes CIDR entries, and curl's
    -x "" is direct. SYSHTTP_PROXY_AUTH goes to SYSHTTP_PROXY only and
    HTTP_PROXY_AUTH to the machine's proxies only.

  • A list from the system (PROXY a; PROXY b; DIRECT) is tried in order by the
    session, httpx and aiohttp (proxylib's failover), replacing the probe of
    the entry above; urllib takes the first reachable entry. No proxy
    connecting and no DIRECT is the client's own connection error;
    NoReachableProxyError is proxylib's.

  • The session's ProxyAdapter is proxylib's ProxyMapAdapter with the
    system's TLS; adapter.route(url) returns the entries it will try.
    proxies= given to new_session are requests' own, with no syshttp
    credential added.

  • syshttp.autopatch routes through proxylib.patch() and patches TLS only;
    requests' TLS is a pool hook on every HTTPAdapter. A caller's own
    proxies and trust_env = False still win over syshttp (proxylib 2.0.2's
    rule). aiohttp is not offered
    URL-prefix gateways. New target pip (pip._vendor.requests, proxylib
    2.0.1's pip target): pip's adapters route through the map unless pip is
    given --proxy or --no-proxy-env; pip keeps its own TLS context.

  • The jar constructors (FileCookieJar, FileTokenJar) no longer create
    their directory; it is created at the first write.

  • A session returns the last response when a status in status_forcelist
    outlasts its retries, as plain requests does; it used to raise
    requests.exceptions.RetryError, which the fetch helpers (-> (status, text)) passed on. The *_auto helpers take new_session's options as
    **session_options (every keyword they had still works) and their
    timeout may be a (connect, read) pair.

  • Every dependency's floor is its newest release that supports Python 3.9:
    the session extra requires requests>=2.32.5 and urllib3>=2.6.3 (was
    requests>=2.25), and sslcontext>=2.1.0. Below requests 2.32.3 a caller's
    ssl_context was silently ignored, urllib3 1.26 raises a bare ValueError
    for an https IP-literal host, and urllib3 up to 2.2 leaks th...

Read more