Fix BigQueryInsertJobOperator error handling in deferrable mode

[Mohammed-Karim226]

Apache Airflow version

main (development)

If "Other Airflow 2 version" selected, which one?

I am using Apache Airflow version 2.6.2.

What happened?

When using the BigQueryInsertJobOperator in deferrable mode, if the job fails without being deferred, the task does not raise an exception or fail as expected. Instead, it continues execution without proper error handling, leading to incorrect task states and potential data inconsistencies.

What you think should happen instead?

The task should raise an exception and fail immediately if the job encounters an error in non-deferred mode, ensuring proper error handling and task state management.

How to reproduce

1. Set up a DAG with the BigQueryInsertJobOperator in deferrable mode.
2. Configure the operator to execute a job that will fail (invalid SQL query).
3. Trigger the DAG and observe the task behavior.

Operating System

Ubuntu 20.04 LTS

Versions of Apache Airflow Providers

8.10.0

Deployment

Docker-Compose

Deployment details

• Airflow version: 2.6.2
• Docker Compose version: 2.17.2
• Kubernetes version: N/A
• Custom configurations: None

Anything else?

This issue occurs every time a non-deferred job fails in the BigQueryInsertJobOperator. Below are the relevant logs:

log
[2025-01-28T12:34:56.789Z] ERROR - Task failed but did not raise an exception. Task state: SUCCESS

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[eladkal]

@Mohammed-Karim226 You have track record of posting spam issues in this project and in others. I am closing this issue for the same reason.

If you are a real person with real intent to improve the project please interact like people do. We do not appreciate wasting our time. If you are a real Google user with a real problem do the extra mile to prove this is a real issue. Show actual logs with images proving the failure show the actual real code you ran to face the error and present it both in Airflow and Google cloud env.