Skip to content

airflow-ctl: migrate to httpx 1.x (after rewrite) #65609

@potiuk

Description

@potiuk

Context

httpx 1.0 (currently `1.0.dev3` on PyPI) is a ground-up rewrite rather than a normal major bump. Inspecting `httpx==1.0.dev3`:

  • The entire public exception hierarchy is gone — only `ProtocolError` remains. No more `HTTPError`, `RequestError`, `TransportError`, `HTTPStatusError`, `TimeoutException`, `ConnectError`, `ReadError`, `WriteError`, `ProxyError`, `TooManyRedirects`, `DecodingError`, `StreamError`, `InvalidURL`, etc.
  • `Response.raise_for_status()` → needs a replacement check API.
  • `AsyncClient` / `Client` / `Transport` / `Auth` surface is redesigned.
  • Many new primitives (`serve_http`, `run`, `DuplexStream`, `HTTPParser`, `ConnectionPool`) — it ships as both a client and a server.

Current airflow-ctl usage

`airflow-ctl/src/airflowctl/api/client.py` and `airflow-ctl/src/airflowctl/api/operations.py` use:

  • `httpx.Client`, `httpx.AsyncClient`
  • `httpx.HTTPStatusError` (including a subclass `ServerResponseError(httpx.HTTPStatusError)`)
  • `httpx.ConnectError`, `httpx.ReadError` in retry/error-handling logic
  • `Response.raise_for_status()`
  • `httpx.MockTransport` in tests (`airflow-ctl/tests/airflow_ctl/api/test_operations.py`)

Interim mitigation

#65607 caps `httpx<1.0` so users who pass `--pre` / `--prerelease=allow` do not pull the rewrite and get a crashing install. That unblocks 0.1.4rc3 but does not address the migration.

What this issue tracks

Migrating airflow-ctl to httpx 1.x once it stabilizes:

  • Audit `httpx` API usage in `api/client.py`, `api/operations.py`, and tests.
  • Design replacements for `ServerResponseError(httpx.HTTPStatusError)` and the retry/error-handling paths (likely a custom exception hierarchy that is not tied to an httpx public one).
  • Port `httpx.MockTransport`-based tests to whatever 1.x exposes for test-time HTTP mocking.
  • Update `httpx>=0.27.0,<1.0` → `>=1.0` in `airflow-ctl/pyproject.toml` once the rewrite is released and adopted.
  • Verify `airflowctl` remote-CLI workflows against the 1.x client surface (auth flows, streaming downloads for logs, timeout semantics).

Acceptance

`airflowctl` installs cleanly with `pip install --pre apache-airflow-ctl` on httpx 1.x and all existing airflow-ctl tests pass.


Opened as part of the 0.1.4rc3 release prep — see #65497 for rc2 testing feedback that surfaced the incompatibility.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions