Use async DB session for Execution API task-instance heartbeat

[Dev-iL]

Related:

• [POC][WIP] Async SQLAlchemy sessions in Airflow #36504
• Migrate API endpoints from sync to async DB sessions #67799

What & why

Converts the Execution API ti_heartbeat route from the synchronous SessionDep to the async AsyncSessionDep, adopting the async metadata engine that already ships in Airflow 3.x (create_async_engine, async_sessionmaker, AsyncSessionDep, create_session_async).

Heartbeat is the highest-QPS write path in the system (one call per running task per heartbeat interval, per worker), so it is a meaningful first production route to run on the async engine. The goal of this PR is behavioral parity with zero regression — not a throughput change. It follows the same low-blast-radius conversion pattern already merged for GET /execution/variables/keys.

What changed

• Route (execution_api/routes/task_instances.py): ti_heartbeat is now async def and takes AsyncSessionDep; the fast-path UPDATE, the slow-path SELECT … FOR UPDATE (.one()), the Task-Instance-History existence scalar, and the final UPDATE are awaited. No explicit commit() is added — the async dependency commits on success and rolls back (releasing the FOR UPDATE row lock) on exception, mirroring the sync dependency exactly. synchronize_session=False and with_for_update() are unchanged.
• Tests (versions/head/test_task_instances.py):
  • The two tests that intercept the fast-path UPDATE now patch sqlalchemy.ext.asyncio.AsyncSession with an async interceptor (the route now awaits AsyncSession.execute); assertions are unchanged.
  • Added a reconfigure_async_db_engine autouse fixture to TestTIHealthEndpoint. The async engine binds its connection pool to the event loop that created it, while the test harness builds a fresh app and event loop per test; without this, a pooled connection from a prior test's closed loop is reused and fails (attached to a different loop). This is the same workaround already used by TestWaitDagRun.
• Docs: documented the asyncpg + transaction-mode PgBouncer prepared-statement caveat on the sql_alchemy_connect_args_async config reference and in the PgBouncer section of the database setup guide.
• Newsfragment (improvement): notes that the heartbeat endpoint now uses the async engine and that the API server may hold both the sync and async connection pools concurrently, so operators should budget DB max_connections for both.

Behavioral parity

No change to the endpoint contract: same 204 success, same 404 / 409 (running-elsewhere / not-running) / 410 (cleared-and-archived) responses with identical detail payloads, same request signature. No Execution API version bump. The pre-existing heartbeat tests pass unchanged — that byte-identical pass is the parity proof.

Testing

Heartbeat suite (-k heartbeat, 12 tests covering success, fast path, slow-path fallback, 404, 409, 410, and the rowcount == 0 / unknown-rowcount fall-through branches) passes on all three supported backends:

Backend	Async driver	Result
SQLite	aiosqlite	12/12 ✅
Postgres	asyncpg	12/12 ✅
MySQL	aiomysql	12/12 ✅

ruff and mypy (airflow-core) are clean.

Operational notes

• Connection budget: heartbeat moving to the async engine means a busy API server can hold both the sync pool (for the remaining sync routes) and the async pool concurrently. Size DB max_connections accordingly (called out in the newsfragment).
• PgBouncer (transaction pooling): asyncpg uses server-side prepared statements, which break under transaction-mode PgBouncer. This is documented as a deployment-configuration requirement here; shipping pgbouncer-safe async-engine defaults is deferred and tracked in #<tracking-issue>.

Note to reviewers

A cosmetic Event loop is closed message is logged during session teardown when the async engine is disposed (dispose_orm closing the async engine synchronously). It appears on all three async drivers, is pre-existing engine disposal behavior unrelated to this change, and every run exits 0. Tidying async-engine disposal is out of scope for this PR.

Engine-default hardening for asyncpg/PgBouncer will be tracked in a dedicated issue to be opened soon.

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

Generated-by: Claude Opus 4.8 following the guidelines

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[ashb]

I think instead of just noting this (which will get ignored/missed by many users) we should change the default async to psycopg3. The only reason we had it set up as asyncpg was because we were stuck on SQLA1.4, but you've already fixed that :)

[Dev-iL]

Do we not want to keep asyncpg for the better performance? I did a benchmark recently and this is still the fastest driver.

[ashb]

I wonder what impact re-creating the connection many times will have on test durations? (Not much for this case, but as we move more things to async?)

Should we look at changing the default event-loop scoping maybe? That also has draw backs for sure.

[ashb]

Nice one.

[ashb]

• Connection budget: heartbeat moving to the async engine means a busy API server can hold both the sync pool (for the remaining sync routes) and the async pool concurrently. Size DB max_connections accordingly (called out in the newsfragment).
• PgBouncer (transaction pooling): asyncpg uses server-side prepared statements, which break under transaction-mode PgBouncer. This is documented as a deployment-configuration requirement here; shipping pgbouncer-safe async-engine defaults is deferred and tracked in #.

I think the agent made these up :)