Skip to content

[AIRFLOW-5610] Add ability to specify multiple objects to copy in GCSToGCSOperator#7728

Merged
potiuk merged 6 commits intoapache:masterfrom
ephraimbuddy:gcs
Mar 18, 2020
Merged

[AIRFLOW-5610] Add ability to specify multiple objects to copy in GCSToGCSOperator#7728
potiuk merged 6 commits intoapache:masterfrom
ephraimbuddy:gcs

Conversation

@ephraimbuddy
Copy link
Contributor

@ephraimbuddy ephraimbuddy commented Mar 15, 2020

  • Rewrote the GCSToGCSOperator class to accept a list of objects to copy rather than a single object.
  • Added tests for the GCSToGCSOperator.
  • Made some change in documentation

Issue link: AIRFLOW-5610

Make sure to mark the boxes below before creating PR: [x]

  • Description above provides context of the change
  • Commit message/PR title starts with [AIRFLOW-NNNN]. AIRFLOW-NNNN = JIRA ID*
  • Unit tests coverage for changes (not needed for documentation changes)
  • Commits follow "How to write a good git commit message"
  • Relevant documentation is updated including usage instructions.
  • I will engage committers as explained in Contribution Workflow Example.

* For document-only changes commit message can start with [AIRFLOW-XXXX].


In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

@boring-cyborg boring-cyborg bot added area:docs provider:google Google (including GCP) related issues labels Mar 15, 2020
Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! just a few nits.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be List [str] or str

But maybe it would be better to keep both source_object and source_objects parameters (source_object = str, source_objects=List[str]) and check that only one is set?

I think that would be still backwards compatible, but more "accurate" with naming.

But I leave it up to you - I am ok with both approaches (as long as we have correct type).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. I'll take that approach of source_object and source_objects. Thanks a lot

@codecov-io
Copy link

codecov-io commented Mar 17, 2020

Codecov Report

Merging #7728 into master will decrease coverage by 27.09%.
The diff coverage is 13.33%.

Impacted file tree graph

@@             Coverage Diff             @@
##           master    #7728       +/-   ##
===========================================
- Coverage   86.99%   59.90%   -27.10%     
===========================================
  Files         915      915               
  Lines       44198    44228       +30     
===========================================
- Hits        38451    26494    -11957     
- Misses       5747    17734    +11987     
Impacted Files Coverage Δ
...low/providers/google/cloud/operators/gcs_to_gcs.py 44.44% <13.33%> (-51.21%) ⬇️
airflow/providers/amazon/aws/hooks/kinesis.py 0.00% <0.00%> (-100.00%) ⬇️
airflow/providers/apache/livy/sensors/livy.py 0.00% <0.00%> (-100.00%) ⬇️
airflow/providers/google/suite/hooks/sheets.py 0.00% <0.00%> (-100.00%) ⬇️
airflow/providers/amazon/aws/sensors/redshift.py 0.00% <0.00%> (-100.00%) ⬇️
airflow/providers/postgres/operators/postgres.py 0.00% <0.00%> (-100.00%) ⬇️
airflow/providers/microsoft/azure/operators/adx.py 0.00% <0.00%> (-100.00%) ⬇️
...irflow/providers/amazon/aws/hooks/batch_waiters.py 0.00% <0.00%> (-100.00%) ⬇️
...ow/providers/amazon/aws/sensors/cloud_formation.py 0.00% <0.00%> (-100.00%) ⬇️
...w/providers/apache/hive/operators/mysql_to_hive.py 0.00% <0.00%> (-100.00%) ⬇️
... and 305 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 49998ed...1813ca4. Read the comment docs.

@potiuk
Copy link
Member

potiuk commented Mar 18, 2020

Thanks @ephraimbuddy !

@potiuk potiuk merged commit 60fdbf6 into apache:master Mar 18, 2020
@ephraimbuddy ephraimbuddy deleted the gcs branch March 18, 2020 19:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants