Enable asynchronous job submission in BigQuery Insert Job #21385
Conversation
boring-cyborg
bot
added
area:providers
provider:google
There was a problem hiding this comment.
@phanikumv I think we should return {"job_id": self.job_id} from execute so it could be used in a sensor. can you do that and add a test for that behavoir?
thanks
disregard; i was looking at wrong operator in the file; it's already returning job id |
phanikumv
requested review from
ashb,
kaxil,
potiuk and
XD-DENG
as code owners
phanikumv
force-pushed
the
async-bigquery-insert-job
branch
from
1955570
to
4daaafa
Compare
phanikumv
force-pushed
the
async-bigquery-insert-job
branch
from
25ecef2
to
0d27f2c
Compare
github-actions
bot
added
the
okay to merge
|
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease. |
- Add is_async flag to the insert_job method - When is_async is True, the execution won't wait till the job results are available. - By default, the job execution will wait till job results are available.
- Pass is_async flag through a pytest fixture with True or False values - Modify test_insert_job method to accept is_async as parameter - Change is_async flag from Optional[bool] to bool
- Add is_async flag to the insert_job method - When is_async is True, the execution won't wait till the job results are available. - By default, the job execution will wait till job results are available.
dstandish
force-pushed
the
async-bigquery-insert-job
branch
from
0d27f2c
to
64859a0
Compare
|
@phanikumv why this |
|
@lwyszomi my thought process was that for async job execution requirements, we can make use of the BigQueryInsertJobOperatorAsync , which executes the insert_job method with nowait set to True. |
cc @dstandish @kaxil
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.