DataprocCreateClusterOperator: Fix non-deferrable reconciliation and handle deletion during creation#61951
Open
SameerMesiah97 wants to merge 1 commit intoapache:mainfrom
Conversation
boring-cyborg
bot
added
area:providers
provider:google
Contributor
Author
Yes. I will fix it. |
…iliation runs after creation completes. – Extract reconciliation logic into `_reconcile_cluster_state()` – Ensure DELETING state waits for deletion and re-creates the cluster – Ensure CREATING state is fully reconciled before returning – Handle STOPPED state via restart path – Raise explicit exception if cluster is not found after LRO completion – Return reconciled cluster to avoid stale state Update and extend unit tests to cover reconciliation scenarios in the non-deferrable path (CREATING, DELETING, STOPPED, ERROR, and timeout cases).
SameerMesiah97
force-pushed
the
61947-DataprocCreateClusterOperator-Recon-Fix
branch
from
613a843 to
747bf37
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This change refactors the
DataprocCreateClusterOperator.executemethod to ensure cluster state reconciliation is consistently applied in non-deferrable mode.After submitting the cluster creation request and waiting for the long-running operation (LRO) to complete, the operator now explicitly fetches the current cluster state and passes it through a dedicated
_reconcile_cluster_statemethod before returning success.The reconciliation logic, previously embedded inline in
execute, has been consolidated into_reconcile_cluster_state. This method handles clusters inCREATING,DELETING, andSTOPPEDstates by waiting, recreating, or restarting as appropriate.Rationale
The operator docstring specifies that when
use_if_exists=True, the operator should:CREATINGDELETINGERRORstate appropriatelyAlthough state-handling logic existed, the non-deferrable execution path previously returned immediately after the create LRO completed, preventing the existing reconciliation logic from being triggered in certain scenarios (e.g. cluster transitioning to
DELETINGduring creation).This change ensures the pre-existing reconciliation behavior is executed consistently, aligning runtime behavior with the documented contract.
Notes
NotFoundhandling after the Long-Running Operation (LRO) completes to surface a clearAirflowExceptionif the cluster was deleted before its state could be reconciled.Tests
Unit tests have been added to cover reconciliation scenarios:
AirflowExceptionwhen the cluster remains inDELETINGstate and deletion is not triggered.delete_on_error=True.Existing tests have been updated to align with the new reconciliation flow and state handling behavior.
Backwards Compatibility
There is no intended change to the operator’s public contract. The implementation now consistently executes the previously defined reconciliation logic in non-deferrable mode.
Closes: #61947