Skip to content

Add support for non-preemptible secondary workers in Dataproc operator#10714

Closed
medb wants to merge 1 commit intoapache:masterfrom
medb:patch-1
Closed

Add support for non-preemptible secondary workers in Dataproc operator#10714
medb wants to merge 1 commit intoapache:masterfrom
medb:patch-1

Conversation

@medb
Copy link

@medb medb commented Sep 3, 2020

No description provided.

@boring-cyborg boring-cyborg bot added the provider:google Google (including GCP) related issues label Sep 3, 2020
@boring-cyborg
Copy link

boring-cyborg bot commented Sep 3, 2020

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:

  • Pay attention to the quality of your code (flake8, pylint and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://apache-airflow-slack.herokuapp.com/

@mik-laj mik-laj requested a review from turbaszek September 3, 2020 21:27
@turbaszek
Copy link
Member

@medb thanks for your contribution! The ClusterGenerator is here only to preserver backward-compatibility. Currently, the DataprocCreateCluster operator accepts a dictionary that describes the cluster. In this way, we can easily cover all Dataproc cluster configurations without tones of additional code. So I would suggest to use the backports:
https://github.com/apache/airflow/tree/master/airflow/providers/google#installation

@medb
Copy link
Author

medb commented Sep 4, 2020

Thank you for clarification, in this case I will close this PR because it's already possible to use DataprocCreateCluster operator to specify secondary workers preemptibility.

@medb medb closed this Sep 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants