Skip to content

Keep asyncpg installed by default in the Postgres provider#69690

Merged
shahar1 merged 1 commit into
apache:mainfrom
shahar1:fix-postgres-asyncpg-default-dep
Jul 10, 2026
Merged

Keep asyncpg installed by default in the Postgres provider#69690
shahar1 merged 1 commit into
apache:mainfrom
shahar1:fix-postgres-asyncpg-default-dep

Conversation

@shahar1

@shahar1 shahar1 commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Problem

Postgres provider 7.0.0 (currently in RC as 7.0.0rc1, #69526 / merged #69089) made psycopg3 the default async metadata-DB driver and, as part of that, moved asyncpg from a default dependency to the opt-in [asyncpg] extra.

The psycopg-async default and its postgresql+asyncpg:// fallback only exist on Airflow 3.4.0+ (airflow-core settings.py::_get_async_conn_uri_from_sync). Every currently released 3.x core (3.0–3.3), which this provider still supports (apache-airflow>=2.11.0), derives the async metadata-DB URL as postgresql+asyncpg:// unconditionally and has no psycopg fallback.

So upgrading the Postgres provider to 7.0.0 on any released 3.x core (or a 3.4.0 deployment that pinned an older core) leaves the async metadata-DB engine unable to load its driver:

sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:postgresql.asyncpg

This is a regression against cores the provider claims to support, and a reason to hold postgres 7.0.0rc1.

Fix

Keep asyncpg as a default dependency of the Postgres provider until its minimum supported Airflow is 3.4.0 (i.e. until every supported core knows the psycopg_async fallback). psycopg3 remains the default async driver on 3.4.0+; asyncpg is simply still present so older supported cores keep working. The removal is gated on the floor bump and noted at the workaround site.

  • pyproject.toml: add asyncpg>=0.30.0 back to default dependencies (comment explains the constraint + tracking issue).
  • changelog.rst: correct the 7.0.0 note — asyncpg stays installed by default; the psycopg_async switch is 3.4.0+ behavior only.
  • README.rst / docs/index.rst / uv.lock: regenerated by prek hooks.

The eventual removal (converge on psycopg3-only) is tracked at #68453.

related: #69526
related: #68453

Test plan

Dependency/packaging + changelog change; no Python behavior to unit-test (the affected async-URL derivation lives in airflow-core, unchanged here). Verified: prek Update dependencies for providers, Sync provider README.rst Requirements table with pyproject.toml, Update uv.lock, and changelog-format hooks all pass.

Note: two related core hardening items remain as separate follow-ups (not required to unblock the RC): core _USE_PSYCOPG3 gates on find_spec("psycopg") only, missing the SQLAlchemy-2 guard the provider hook has; and the derived async URL is not validated against installed drivers, so failures surface as an opaque NoSuchModuleError.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.8)

Generated-by: Claude Code (Opus 4.8) following the guidelines

Postgres provider 7.0.0 moved asyncpg from a default dependency to an
opt-in extra when it made psycopg3 the default async metadata-DB driver.
That default only exists on Airflow 3.4.0+, whose async URL derivation can
fall back to asyncpg. Airflow cores older than 3.4.0 — all currently
released 3.x lines, which this provider still supports (apache-airflow>=2.11.0)
— derive the async metadata-DB URL as postgresql+asyncpg:// unconditionally
and have no psycopg fallback, so dropping asyncpg leaves their async engine
unable to load its driver.

Keep asyncpg as a default dependency until the minimum supported Airflow is
3.4.0, so upgrading the provider does not break the async metadata database
on the cores it still claims to support.
@shahar1 shahar1 force-pushed the fix-postgres-asyncpg-default-dep branch from 7fa1fb2 to be88220 Compare July 10, 2026 06:01

@Dev-iL Dev-iL left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for opening this. The reasoning is sound, the implementation looks correct.

@shahar1 shahar1 requested review from eladkal, potiuk and vincbeck July 10, 2026 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants