Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#16691 Providing more information in docs for DataprocCreateCluster operator migration #19446

Conversation

pateash
Copy link
Contributor

@pateash pateash commented Nov 6, 2021

closes: #16911

Add meaningful description above

I am proposing to add a bit more information in the documentation regarding usage of DataprocClusterCreateOperator() as per refactoring in #6371, users find it a bit difficult to generate CLUSTER_CONFIG.

@pateash pateash requested a review from eladkal November 7, 2021 18:25
@pateash pateash changed the title #16691 fix passing metadata to dataproc cluster create operator #16691 Providing more information in docs for DataprocCreateCluster operator migration Nov 7, 2021
@pateash
Copy link
Contributor Author

pateash commented Nov 7, 2021

image

@github-actions github-actions bot added the okay to merge It's ok to merge this PR as it does not require more tests label Nov 15, 2021
@github-actions
Copy link

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

@potiuk potiuk merged commit 48f228c into apache:main Nov 15, 2021
@pateash pateash deleted the #16691-fix-passing-metadata-to-DataprocClusterCreateOperator branch November 16, 2021 04:49
@pateash pateash restored the #16691-fix-passing-metadata-to-DataprocClusterCreateOperator branch November 27, 2021 02:17
@guptaneha0908
Copy link

guptaneha0908 commented Dec 21, 2021

@pateash I am facing similar issue with DataprocCreateClusterOperator as well(Airflow 2+) . Also I see metadata field in DataprocCreateClusterOperator is of type Sequence[Tuple[str, str]] but the ClusterGenerator() has type dict for metadata field.
This seems to contradict. I see this branch is merged. Also as per my knowledge ClusterGenerator() is deprecated method. Could you let me know the fix for newer Operator

@pateash pateash deleted the #16691-fix-passing-metadata-to-DataprocClusterCreateOperator branch December 23, 2021 00:50
@pateash
Copy link
Contributor Author

pateash commented Dec 23, 2021

@pateash I am facing similar issue with DataprocCreateClusterOperator as well(Airflow 2+) . Also I see metadata field in DataprocCreateClusterOperator is of type Sequence[Tuple[str, str]] but the ClusterGenerator() has type dict for metadata field. This seems to contradict. I see this branch is merged. Also as per my knowledge ClusterGenerator() is deprecated method. Could you let me know the fix for newer Operator

@guptaneha0908 ClusterGenerator() has been kept for easy migration, ideally you should have our CLUSTER_CONFIG as dictionary. You can avoid using ClusterGenerator by creating CLUSTER_CONFIG from it once and then storing that dictionary and passing to DataProcClusterCreateOperator going forward.

@jedcunningham jedcunningham added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Apr 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) kind:documentation okay to merge It's ok to merge this PR as it does not require more tests provider:google Google (including GCP) related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Error in passing metadata to DataprocClusterCreateOperator
7 participants