Refactor: Convert BigQueryAsyncHook.get_records into an async method to prevent event loop blocking

[anushkagupta200615-jpg]

Under which category would you file this issue?

Providers

Apache Airflow version

main

What happened and how to reproduce it?

Description
In the Google Provider's BigQueryAsyncHook, the get_records method is currently implemented as a standard synchronous function rather than a native async def method.

Because get_records is synchronous, whenever a deferrable operator or trigger uses this method to process and format large datasets returned from BigQuery, it holds up the Python thread. This degrades the performance of the Airflow Triggerer process, as other async tasks cannot run while the records are being processed.

There is an explicit # TODO: Convert get_records into an async method at line 2440 in airflow/providers/google/cloud/hooks/bigquery.py.

Actual Behaviour
When a deferrable BigQuery operator completes its query and calls BigQueryAsyncHook.get_records to fetch and parse the results, the method executes synchronously. If the query result is very large, the data processing consumes CPU time continuously on the main thread, blocking the Triggerer's asyncio event loop and delaying the execution of other active triggers.

Steps to Reproduce

1. Create an Airflow DAG that uses a deferrable BigQuery operator (e.g., BigQueryInsertJobOperator with deferrable=True).
2. Run a query that returns a massive dataset (millions of rows).
3. The operator will defer to the Triggerer. Once the job finishes, the Triggerer will execute BigQueryAsyncHook.get_records to fetch the data.
4. During the execution of get_records, observe the Triggerer logs and performance metrics. You will notice that the event loop is blocked and other deferred tasks are delayed from firing until the synchronous data parsing is fully complete.

What you think should happen instead?

Proposed Behaviour
BigQueryAsyncHook.get_records should be an async def method. During heavy data processing or when paginating through results, it should yield control back to the event loop (e.g., via await asyncio.sleep(0)) so that the Triggerer remains non-blocking and highly concurrent.

Solution

1. Refactor get_records in providers/google/src/airflow/providers/google/cloud/hooks/bigquery.py to be an async def method.
2. If the parsing logic is CPU-bound, introduce await asyncio.sleep(0) in the loop chunks to yield control back to the event loop.
3. Update any triggers or deferrable operators in the Google provider that call BigQueryAsyncHook.get_records to properly await the newly asynchronous method (e.g., records = await hook.get_records(...)).

Operating System

Windows 11

Deployment

None

Apache Airflow Provider(s)

google

Versions of Apache Airflow Providers

main

Official Helm Chart version

Not Applicable

Kubernetes Version

No response

Helm Chart configuration

No response

Docker Image customizations

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.