feat: add idle/auto stop TTLs and master/worker instance flexibility …

[sravani-bobbala]

…policies to Dataproc cluster configuration

This pull request adds support for configuring Instance Flexibility Policy and Lifecycle Configuration (specifically idle stop TTL and auto stop time) in the Dataproc ClusterGenerator.

Description

• Instance Flexibility Policy: Added support for specifying instance flexibility policies for both master and worker nodes. This allows users to provide a list of machine types that Dataproc can choose from, improving resource availability and cost optimization.
• Lifecycle Configuration: Added support for idle_stop_ttl and auto_stop_time to automatically stop clusters based on inactivity or a specific schedule.
  These changes allow for more flexible and cost-effective Dataproc cluster management directly from Airflow.
  Unit tests have been added to verify that these configurations are correctly generated.

---

Was generative AI tooling used to co-author this PR?

• Yes — Antigravity
  Generated-by: Antigravity following the guidelines

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[potiuk]

@sravani-bobbala A few things need addressing before review — see our Pull Request quality criteria.

• ❌ Pre-commit / static checks: The Static checks job is failing. Static-check failures are deterministic and need a code fix — re-running CI won't help. Run prek run --from-ref main --stage pre-commit locally to reproduce, then push the fix.

No rush.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[potiuk]

@sravani-bobbala A few things need addressing before review — see our Pull Request quality criteria.

Issues found:

• ❌ Pre-commit / static checks: CI image checks / Static checks is failing. Run prek run --from-ref main --stage pre-commit locally and fix anything that flags. See the static-checks docs.

What to do next:

• Push a fix for the static-check failure.

No rush — take your time. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the Airflow Slack.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.