Airflow triggered the upstream task even the downstream task not executed completely.

[ashish1471]

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

Hey Team,
I am not sure where I am doing wrong. I am running a DAG and I have set retries to 5. My expectation is that, If a task got failed, it will retry 5 times and if it doesn't get pass it will mark it as failed but contrary to that, the task marked as success and triggered the upstream task, despite it has not done all the updates required in that particular task to get complete. Am I missing any required parameter? Thanks.

Note- I am using dataProc cluster and updating records in BQ table and it has limitation of 1500 updates/table/day. That limit got exceeded, but airflow didn't bother to mark it failed, rather it has triggered upstream task. Any fix which we can use? so that the task would have failed even if there is limit/quota exceed issue?

Error- 403 BQ limit exceeded.

What you think should happen instead

After limit got exceeded. It should mark the task failed.

How to reproduce

You can run a BQ task on dataproc cluster, set retry to 5 and make updates in BQ table. BQ limitation is 1500 updates/table/day. Try to make around 2000 updates in a single table/day. And see if that task got failed or is it skip that and pass on upstream task. (Do check your BQ table that all your updated records are there or not).

Please let us know what we can do to handle such scenario?

Operating System

windows

Versions of Apache Airflow Providers

No response

Deployment

Composer

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[Taragolis]

1. Can you share the DAG code? You can redact some sensitive values or something that you don't want to expose to the public. Without this it would be problematic reproduce.
2. Which version of Airflow do you use? Did you tried latest version?
3. Which version of Google Provider do you use?
4. Do you use specific Airflow configuration?
5. What Database backend for Airflow do you use?
6. Due to this information, Operating System: windows, do you run Airflow Cluster under WSL2?

[ashish1471]

Hey Taragolis,
Thanks for the help. I digged further on this issue. I got to know that "403 Quota exceeded" coming from Bigquery is coming as an INFO rather than an ERROR; that's why it started executing other tasks despite my table not got updated. Can you tell me how can I tell Airflow that "403 Quota exceeded coming from BQ API" is an error and it should fail the task?

LOG- "INFO: write to table failed: Reason: 403 limit error due to append/update limit exceeded"-

[potiuk]

I think if that's the case - then you need to provide all the details here wiht the log of when exactly and how the quota has been exceeded. Idally you could also upgradet to latest version of the provider and see if it is fixed there. and if not report the exact stack trace from where you get the error. It's very difficult to act on a statement that there is INFO or ERROR without knowing the details of where exactly it's been raised and how. This could be good feedback for @bhirsz @lwyszomi and others who work on the operators.

This looks like a case that is not handled welll and you will need to upgrade anyway to get it fixed - when we manage to release a new version. But having more info is the prerequisite to it.

[github-actions]

This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.

[github-actions]

This issue has been closed because it has not received response from the issue author.