Skip to content

[AIRFLOW-6102] [AIP-21] Rename Dataproc operators#7151

Merged
turbaszek merged 1 commit intoapache:masterfrom
PolideaInternal:AIRFLOW-6102-Rename-dataproc-service
Jan 13, 2020
Merged

[AIRFLOW-6102] [AIP-21] Rename Dataproc operators#7151
turbaszek merged 1 commit intoapache:masterfrom
PolideaInternal:AIRFLOW-6102-Rename-dataproc-service

Conversation

@michalslowikowski00
Copy link
Contributor

@michalslowikowski00 michalslowikowski00 commented Jan 13, 2020

PR contains changes regarding AIP-21 (renaming GCP operators and hooks):

  • renamed GCP modules
  • adde deprecation warnings to the contrib modules
  • fixed tests
  • updated UPDATING.md

Issue link: AIRFLOW-6102

  • Description above provides context of the change
  • Commit message/PR title starts with [AIRFLOW-NNNN]. AIRFLOW-NNNN = JIRA ID*
  • Unit tests coverage for changes (not needed for documentation changes)
  • Commits follow "How to write a good git commit message"
  • Relevant documentation is updated including usage instructions.
  • I will engage committers as explained in Contribution Workflow Example.

* For document-only changes commit message can start with [AIRFLOW-XXXX].


In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

@boring-cyborg boring-cyborg bot added the provider:google Google (including GCP) related issues label Jan 13, 2020
@codecov-io
Copy link

codecov-io commented Jan 13, 2020

Codecov Report

Merging #7151 into master will decrease coverage by 0.24%.
The diff coverage is 96.87%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7151      +/-   ##
==========================================
- Coverage   85.28%   85.03%   -0.25%     
==========================================
  Files         707      707              
  Lines       39255    39347      +92     
==========================================
- Hits        33478    33459      -19     
- Misses       5777     5888     +111
Impacted Files Coverage Δ
...ders/google/cloud/example_dags/example_dataproc.py 0% <0%> (ø) ⬆️
...rflow/providers/google/cloud/operators/dataproc.py 88.13% <100%> (ø) ⬆️
airflow/contrib/hooks/gcp_dataproc_hook.py 100% <100%> (ø) ⬆️
airflow/contrib/operators/dataproc_operator.py 100% <100%> (ø) ⬆️
airflow/kubernetes/volume_mount.py 44.44% <0%> (-55.56%) ⬇️
airflow/kubernetes/volume.py 52.94% <0%> (-47.06%) ⬇️
airflow/kubernetes/pod_launcher.py 45.25% <0%> (-46.72%) ⬇️
airflow/kubernetes/refresh_config.py 50.98% <0%> (-23.53%) ⬇️
...rflow/contrib/operators/kubernetes_pod_operator.py 78.31% <0%> (-20.49%) ⬇️
airflow/contrib/operators/gcs_list_operator.py 100% <0%> (ø) ⬆️
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ab80880...8a7fee8. Read the comment docs.

|airflow.contrib.operators.dataproc_operator.DataProcPigOperator |airflow.gcp.operators.dataproc.DataprocSubmitPigJobOperator |
|airflow.contrib.operators.dataproc_operator.DataProcPySparkOperator |airflow.gcp.operators.dataproc.DataprocSubmitPySparkJobOperator |
|airflow.contrib.operators.dataproc_operator.DataProcSparkOperator |airflow.gcp.operators.dataproc.DataprocSubmitSparkJobOperator |
|airflow.contrib.operators.dataproc_operator.DataProcSparkSqlOperator |airflow.gcp.operators.dataproc.DataprocSubmitSparkSqlJobOperator |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's no need to change names in SubmitXJob operators because those are deperecated anyway by DataprocSubmitJobOperator. WDYT @mik-laj ?

Copy link
Member

@mik-laj mik-laj Jan 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may not be needed, but it is already done and standardizes our code. In my opinion, this is a positive change. We don't need to revert it. Let's go with it.

@turbaszek turbaszek merged commit f4d3e5e into apache:master Jan 13, 2020
@michalslowikowski00 michalslowikowski00 deleted the AIRFLOW-6102-Rename-dataproc-service branch January 26, 2020 08:35
galuszkak pushed a commit to FlyrInc/apache-airflow that referenced this pull request Mar 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants