Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AIRFLOW-5924] Automatically unify bucket name and key in S3Hook #6574

Conversation

feluelle
Copy link
Member

@feluelle feluelle commented Nov 13, 2019

Make sure you have checked all steps below.

Jira

  • My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
    • https://issues.apache.org/jira/browse/AIRFLOW-5924
    • In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.
    • In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal (AIP).
    • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Description

  • Here are some details about my PR, including screenshots of any UI changes:

  • change provide_bucket_name to provide bucket name also for function with keys

  • refactoring

NOTE: This change is not Python 2.7 compatible. Therefore I selected Fix Version 2.0.

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

  • My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it.
    • All the public functions and the classes in the PR contain docstrings that explain what it does
    • If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

@feluelle feluelle force-pushed the feature/AIRFLOW-5924-unify-bucket-name-and-key-s3-hook branch from 2f46122 to 0c29fb1 Compare November 13, 2019 16:59
@feluelle feluelle added the provider:amazon-aws AWS/Amazon - related issues label Nov 13, 2019
@codecov-io
Copy link

codecov-io commented Nov 13, 2019

Codecov Report

Merging #6574 into master will decrease coverage by 0.09%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff            @@
##           master    #6574     +/-   ##
=========================================
- Coverage    86.8%   86.71%   -0.1%     
=========================================
  Files         893      896      +3     
  Lines       42241    43283   +1042     
=========================================
+ Hits        36669    37534    +865     
- Misses       5572     5749    +177
Impacted Files Coverage Δ
airflow/providers/amazon/aws/hooks/s3.py 96.6% <100%> (+0.74%) ⬆️
airflow/kubernetes/volume_mount.py 44.44% <0%> (-55.56%) ⬇️
airflow/kubernetes/volume.py 52.94% <0%> (-47.06%) ⬇️
airflow/kubernetes/pod_launcher.py 47.18% <0%> (-45.08%) ⬇️
...viders/cncf/kubernetes/operators/kubernetes_pod.py 69.69% <0%> (-25.26%) ⬇️
airflow/kubernetes/refresh_config.py 50.98% <0%> (-23.53%) ⬇️
airflow/__init__.py 93.33% <0%> (-6.67%) ⬇️
airflow/jobs/backfill_job.py 90.98% <0%> (-1.13%) ⬇️
...providers/microsoft/azure/hooks/azure_fileshare.py 100% <0%> (ø) ⬆️
...ow/providers/google/cloud/operators/stackdriver.py 100% <0%> (ø)
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3320e43...574aac6. Read the comment docs.

@feluelle feluelle force-pushed the feature/AIRFLOW-5924-unify-bucket-name-and-key-s3-hook branch from 0c29fb1 to 04cd0d7 Compare December 27, 2019 18:20
@feluelle
Copy link
Member Author

feluelle commented Jan 2, 2020

Ready for Review @ashb @potiuk @mik-laj @nuclearpinguin

@feluelle
Copy link
Member Author

feluelle commented Feb 1, 2020

@ashb @potiuk @mik-laj anyone? :)

@feluelle feluelle force-pushed the feature/AIRFLOW-5924-unify-bucket-name-and-key-s3-hook branch from 04cd0d7 to 4376121 Compare February 24, 2020 15:22
- change provide_bucket_name to provide bucket name also for function with keys
- refactoring
@feluelle feluelle force-pushed the feature/AIRFLOW-5924-unify-bucket-name-and-key-s3-hook branch from 4376121 to 574aac6 Compare February 24, 2020 16:24
@feluelle feluelle merged commit 6eaa7e3 into apache:master Feb 25, 2020
galuszkak pushed a commit to FlyrInc/apache-airflow that referenced this pull request Mar 5, 2020
…che#6574)

- change provide_bucket_name to provide bucket name also for function with keys
- refactoring
@feluelle feluelle deleted the feature/AIRFLOW-5924-unify-bucket-name-and-key-s3-hook branch April 30, 2020 07:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
provider:amazon-aws AWS/Amazon - related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants