Skip to content

DataprocDeleteClusterOperator to raise AirflowSkipException if the cluster is already Deleted / Not Found #59854

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
chirodip98 wants to merge 30 commits into from
Closed

DataprocDeleteClusterOperator to raise AirflowSkipException if the cluster is already Deleted / Not Found #59854

chirodip98 wants to merge 30 commits into from

Conversation

@chirodip98
Copy link

@chirodip98 chirodip98 commented Dec 27, 2025 •
edited
Loading

Fixes : (apache/airflow#59812)
In Case of Cluster does not exists , DataprocDeleteClusterOperator will raise AirflowSkipException.

@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Dec 27, 2025
@chirodip98 chirodip98 mentioned this pull request Dec 27, 2025
Open
2 tasks
@chirodip98 chirodip98 changed the title Fix/delete cluster dataproc bug Fix/delete cluster dataproc bug Dec 27, 2025
chirodip98
chirodip98 commented Dec 27, 2025
View reviewed changes
Copy link
Author

@chirodip98 chirodip98 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@potiuk kindly review . I am fairly new to contributing to airflow. Look forward to your guidance and feedback!

@potiuk
Copy link
Member

potiuk commented Dec 27, 2025

@potiuk kindly review . I am fairly new to contributing to airflow. Look forward to your guidance and feedback!

Guidance no.1 : Please avoid pinging individual maintainers - just open PR. and not ping people. When you ping someone, you significantly decrease your chances of getting review by someone else, and when the person you ping is not available or simply does not want to - for any reason - review it, you might wait for a long time for any review (which might happen anyway, but then you can again ping (in general, not individual maintainer) after few days and that will bring your PR back to the top of the pile of PRs.

@potiuk potiuk force-pushed the fix/delete-cluster-dataproc-bug branch from dd10705 to 2f24ab0 Compare December 28, 2025 08:57
chirodip98 and others added 15 commits December 28, 2025 14:48
@chirodip98
Merge branch 'main' into fix/delete-cluster-dataproc-bug
0426aaf
@chirodip98
fix: failing test cases addressed
a36c53e
@chirodip98
Merge branch 'fix/delete-cluster-dataproc-bug' of https://github.com/…
b9258b6 
…chirodip98/airflow-contrib into fix/delete-cluster-dataproc-bug
@chirodip98
fix: failing test cases addressed
e71f067
@chirodip98
fix: Handling scenarios when cluster is non-existent for DataprocDele…
99d1a2c 
…teClusterOperator
@chirodip98
fix: variable renaming fof operation
8f62631
@chirodip98
fix: added type
f9f3dee
@chirodip98
fix:mypy errors fixed for airflow.utils.timezone
fe66a26
@chirodip98
fix: failing test cases addressed
90b5eff
@gopidesupavan @chirodip98
Remove unused assignment dag_version_id from expand_mapped_task (apac…
a3a41e3 
…he#59834)
@jscheffl @chirodip98
Exclude FastAPI 0.128.0 (apache#59856)
eb0efaf
@Dev-iL @chirodip98
DagBag: use Path.relative_to for consistent cross-platform behavior (…
72bebea 
…apache#59785)
@uranusjr @chirodip98
Remove top-level SDK reference in Core (apache#59817)
a80d18d
@chirodip98
fix: Handling scenarios when cluster is non-existent for DataprocDele…
678781a 
…teClusterOperator
@chirodip98
fix: variable renaming fof operation
7a883c0
chirodip98 and others added 4 commits December 28, 2025 16:17
@chirodip98
fix: added type
e156b53
@kunaljubce @chirodip98
Sqlalchemy 2.0 changes for test_scheduler_job.py (apache#59823)
a1db2d3 
* Compatibility updates for sqlalchemy 2.0

* Fixes for review comments

* Fixes for review comments

* More fixes for review comments
@chirodip98
fix: failing test cases addressed
8cf2c36
@chirodip98
chore: conflicts resolved
e5e7d49
chirodip98
chirodip98 commented Dec 28, 2025
View reviewed changes
Copy link
Author

@chirodip98 chirodip98 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need support to run Workfows

@potiuk
Copy link
Member

potiuk commented Dec 28, 2025

I approved it but I thin we need a look from @VladaZakharova and @MaksYermak -> Google team, and - ideally a system test log / screenshots on running this operator using system tests for that operation.

@chirodip98
Copy link
Author

chirodip98 commented Dec 29, 2025 •
edited
Loading

Thanks!
As a System Test I created a bare-minimum Dag which Simulates the scenarios on my local breeze-airflow setup. I will try to add the screenshots for the same, also explore the system tests folder if I can add few test cases.

@chirodip98
Copy link
Author

chirodip98 commented Dec 29, 2025 •
edited
Loading

Scenario 1 : When the Delete operation is performed when Cluster does'nt exists (already-deleted)

image image image

@chirodip98
Copy link
Author

chirodip98 commented Dec 29, 2025 •
edited
Loading

Scenario 2 : When Delete Cluster Operation is performed in defferable mode

image [scenario_2_logs.txt](https://github.com/user-attachments/files/24369267/scenario_2_logs.txt)

@chirodip98
Copy link
Author

chirodip98 commented Dec 29, 2025

Scenario 3 : When Delete operation is perform , but not in deffered mode
image

@chirodip98
Copy link
Author

chirodip98 commented Dec 29, 2025 •
edited
Loading

Scenario 4 : When two Delete actions performed at the same time (raises Exception)
image

@chirodip98 chirodip98 changed the title Fix/delete cluster dataproc bug #59812 : DataprocDeleteClusterOperator to raise AirflowSkipException if the cluster is already Deleted / Not Found Dec 29, 2025
shivannakarthik
shivannakarthik reviewed Jan 2, 2026
View reviewed changes
providers/google/tests/unit/google/cloud/operators/test_dataproc.py

mock_hook.return_value.wait_for_operation.assert_not_called()
assert not mock_defer.called
assert mock_defer.called
Copy link
Contributor

@shivannakarthik shivannakarthik Jan 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this required?

Copy link
Author

@chirodip98 chirodip98 Jan 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes

@chirodip98 chirodip98 force-pushed the fix/delete-cluster-dataproc-bug branch from 0a30780 to a51e6e1 Compare January 2, 2026 08:57
@shahar1 shahar1 changed the title #59812 : DataprocDeleteClusterOperator to raise AirflowSkipException if the cluster is already Deleted / Not Found DataprocDeleteClusterOperator to raise AirflowSkipException if the cluster is already Deleted / Not Found Jan 3, 2026
chirodip98
chirodip98 commented Jan 3, 2026
View reviewed changes
Copy link
Author

@chirodip98 chirodip98 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it okay If I raise a new PR for this issue and close this? I am facing some issues while rebasing. @potiuk

@chirodip98
Copy link
Author

chirodip98 commented Jan 3, 2026

closing this as re-opened
#60083

@chirodip98 chirodip98 closed this Jan 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@potiuk potiuk potiuk approved these changes

+1 more reviewer

@shivannakarthik shivannakarthik shivannakarthik left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

DataprocDeleteClusterOperator fails if cluster was already deleted by DataprocCreateClusterOperator(delete_on_error=True)

8 participants

@chirodip98 @potiuk @shivannakarthik @gopidesupavan @jscheffl @Dev-iL @uranusjr @kunaljubce