Skip to content

Fix Cloud Composer Сreate environment operator to validate final env state after creation#64956

Open
VladaZakharova wants to merge 1 commit intoapache:mainfrom
VladaZakharova:composer-system-test-fix
Open

Fix Cloud Composer Сreate environment operator to validate final env state after creation#64956
VladaZakharova wants to merge 1 commit intoapache:mainfrom
VladaZakharova:composer-system-test-fix

Conversation

@VladaZakharova
Copy link
Copy Markdown
Contributor

Fix Cloud Composer environment creation to succeed only when the environment is actually usable.

Previously, create could rely on stale state from the create long-running operation, and the AlreadyExists path could incorrectly treat an environment in a bad state as success. This PR makes the create operator verify the current environment state and fail unless it is RUNNING.

The system test retry flow is also updated to delete environments left in ERROR before retrying creation.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@VladaZakharova VladaZakharova requested a review from shahar1 as a code owner April 9, 2026 11:20
@boring-cyborg boring-cyborg bot added area:providers kind:documentation provider:google Google (including GCP) related issues labels Apr 9, 2026
@kaxil kaxil requested a review from Copilot April 10, 2026 19:55
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the Google Cloud Composer “create environment” operator to only report success when the environment is actually usable (i.e., in RUNNING), addressing cases where the create LRO or AlreadyExists path could otherwise allow unhealthy environments to pass as successful.

Changes:

  • Re-check Composer environment state via get_environment() after create operation completion and on the AlreadyExists path; fail unless state is RUNNING.
  • Extend unit tests to cover post-operation state validation, AlreadyExists behavior, and deferrable execute_complete validation.
  • Update system test example + docs to add a retry cleanup callback that deletes environments left in ERROR before retrying creation.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
providers/google/src/airflow/providers/google/cloud/operators/cloud_composer.py Adds environment state validation logic for create (sync + deferrable) and AlreadyExists flows.
providers/google/tests/unit/google/cloud/operators/test_cloud_composer.py Adds unit coverage for new state validation behavior and error cases.
providers/google/tests/system/google/cloud/composer/example_cloud_composer.py Adds retry cleanup callback for system-test create retries; configures retries for create tasks.
providers/google/docs/operators/cloud/cloud_composer.rst Documents new “succeed only when RUNNING” behavior and shows retry cleanup example snippet.


if env_state != Environment.State.RUNNING:
raise AirflowException(
"Create environment operation completed, but the Composer environment is not in RUNNING state. "
Comment on lines +83 to +86
def _mock_environment(state: int) -> mock.MagicMock:
environment = mock.MagicMock()
environment.state = state
return environment
Comment on lines +124 to +136
log.info(
"Retry cleanup started for Composer env. project_id=%s region=%s environment_id=%s",
PROJECT_ID,
REGION,
environment_id,
)

try:
environment = hook.get_environment(
project_id=PROJECT_ID,
region=REGION,
environment_id=environment_id,
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers kind:documentation provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants