Skip to content

fix BigQueryInsertJobOperator's return value and openlineage extraction in deferrable mode#40457

Merged
potiuk merged 1 commit intoapache:mainfrom
kacpermuda:fix-bq-deferrable-ol
Jul 2, 2024
Merged

fix BigQueryInsertJobOperator's return value and openlineage extraction in deferrable mode#40457
potiuk merged 1 commit intoapache:mainfrom
kacpermuda:fix-bq-deferrable-ol

Conversation

@kacpermuda
Copy link
Contributor

This PR fixes two things:

  1. Return value of BigQueryInsertJobOperator's execute_complete() (so execution in deferrable mode) will now be an actual BQ job id instead of None. As described in this docs, when deferring: No state will persist, such as local variables or attributes set on self., so assigning value to self.job_id within execute() had no effect and was not actually used in execute_complete().

  2. We are re-assigning the job_id in execute_complete() as an instance attribute, so that we can later use it within OpenLineage method even when in deferrable mode. We are also re-creating any attributes (hook) that may be missing as a result of the state keeping logic mentioned above.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Jun 27, 2024
@kacpermuda kacpermuda force-pushed the fix-bq-deferrable-ol branch from a0c67d2 to e12c4e0 Compare June 27, 2024 14:01
Copy link
Contributor

@shahar1 shahar1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please add some tests?

@kacpermuda kacpermuda force-pushed the fix-bq-deferrable-ol branch from e12c4e0 to dc21a36 Compare July 1, 2024 09:56
@kacpermuda
Copy link
Contributor Author

Added some tests to check that we indeed re-assign the job_id from the event. @shahar1

…on in deferrable mode

Signed-off-by: Kacper Muda <mudakacper@gmail.com>
@kacpermuda kacpermuda force-pushed the fix-bq-deferrable-ol branch from dc21a36 to 1bd08eb Compare July 2, 2024 08:25
@potiuk potiuk merged commit db16eeb into apache:main Jul 2, 2024
@kacpermuda kacpermuda deleted the fix-bq-deferrable-ol branch July 2, 2024 09:30
romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Jul 26, 2024
…on in deferrable mode (apache#40457)

Signed-off-by: Kacper Muda <mudakacper@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants