[AIRFLOW-5816] S3 to snowflake operator #6469

shaikshakeel · 2019-10-30T10:54:14Z

Make sure you have checked all steps below.

Jira

My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5816
- In case you are fixing a typo in the documentation, you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal (AIP).
- In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Description

Here are some details about my PR, including screenshots of any UI changes:

Tests

My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters (not including Jira issue reference)
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"

Documentation

In case of new functionality, my PR adds documentation that describes how to use it.
- All the public functions and the classes in the PR contain docstrings that explain what it does
- If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

codecov-io · 2019-10-30T12:03:30Z

Codecov Report

❗ No coverage uploaded for pull request base (master@bbe2f61). Click here to learn what that means.
The diff coverage is 93.4%.

@@            Coverage Diff            @@
##             master    #6469   +/-   ##
=========================================
  Coverage          ?   85.17%           
=========================================
  Files             ?      756           
  Lines             ?    39721           
  Branches          ?        0           
=========================================
  Hits              ?    33832           
  Misses            ?     5889           
  Partials          ?        0

Impacted Files	Coverage Δ
airflow/providers/snowflake/operators/snowflake.py	`0% <ø> (ø)`
airflow/bin/cli.py	`94.73% <ø> (ø)`
airflow/gcp/hooks/bigquery.py	`91.81% <ø> (ø)`
airflow/utils/log/gcs_task_handler.py	`0% <0%> (ø)`
...ders/google/cloud/example_dags/example_dataproc.py	`0% <0%> (ø)`
airflow/providers/snowflake/hooks/snowflake.py	`0% <0%> (ø)`
airflow/operators/cassandra_to_gcs.py	`64.91% <100%> (ø)`
airflow/contrib/hooks/gcs_hook.py	`100% <100%> (ø)`
airflow/gcp/hooks/gcs.py	`84.98% <100%> (ø)`
airflow/operators/gcs_to_gcs.py	`95.65% <100%> (ø)`
... and 39 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bbe2f61...ce62aa1. Read the comment docs.

OmerJog · 2019-11-05T13:18:34Z

FYI duplicated Jira https://issues.apache.org/jira/browse/AIRFLOW-3031

shaikshakeel · 2019-11-05T13:29:12Z

@OmerJog can you give a pull request for this?

OmerJog · 2019-11-12T08:23:07Z

@shaikshakeel it's just a note for committers to close handle the duplicate jira.

@feluelle can you review?

airflow/contrib/operators/s3_to_snowflake_operator.py

tests/contrib/operators/test_s3_to_snowflake_operator.py

…nowflake_operator # Conflicts: # docs/operators-and-hooks-ref.rst

feluelle

I think for new modules we should directly follow AIP-21 and place the operator directly into the providers package providers/snowflake.

In case of transfer operators where two providers are involved, the transfer operators will be moved to ~~"source"~~ (NOTE it's been changed to "target" in subsequent Update) of the transfer. When there is only one provider as target but source is a database or another non-provider source, the operator is put to the target provider.

@mik-laj @potiuk am I correct?

shaikshakeel · 2019-11-25T14:14:48Z

Hi @feluelle

Is the below path is okay?

providers/snowflake/operators/s3_to_snowflake ?
providers/snowflake/hooks/snowflake ?
providers/snowflake/operators/snowflake ?

OmerJog · 2019-11-25T16:23:09Z

@feluelle This is S3 to snowflake so it should be
/providers/amazon/aws/operators
isn't it?

shaikshakeel · 2019-12-02T11:06:15Z

@feluelle can you confirm this?

feluelle · 2019-12-02T14:27:06Z

I think it belongs to the snowflake provider because..

In case of transfer operators where two providers are involved, the transfer operators will be moved to ~~"source"~~ (NOTE it's been changed to "target" in subsequent Update) of the transfer. When there is only one provider as target but source is a database or another non-provider source, the operator is put to the target provider.

So

providers/snowflake/operators/s3_to_snowflake
providers/snowflake/hooks/snowflake
providers/snowflake/operators/snowflake

is correct imo.

feluelle · 2019-12-02T14:27:53Z

@mik-laj @potiuk please correct me if I am wrong.

shaikshakeel · 2019-12-02T17:45:27Z

@mik-laj @potiuk can you confirm this. I will change accordingly

potiuk · 2019-12-09T16:11:30Z

Hello. I missed that one. Sorry. Yes it should be snowflake ( the operator) hooks should be in s3/snowflake.

…nowflake_operator # Conflicts: # scripts/ci/pylint_todo.txt

airflow/providers/snowflake/hooks/snowflake.py

airflow/providers/snowflake/operators/snowflake.py

docs/operators-and-hooks-ref.rst

Co-Authored-By: Felix Uellendall <feluelle@users.noreply.github.com>

…nowflake_operator

airflow/providers/snowflake/operators/snowflake.py

Co-Authored-By: Felix Uellendall <feluelle@users.noreply.github.com>

- move Snowflake from contrib to providers - fix pylint issues Co-authored-by: Felix Uellendall <feluelle@users.noreply.github.com>

shaikshakeel added 6 commits October 29, 2019 14:20

add s3_to_snowflake_operator

c115658

pylink, flake8, doc issues fixed

ae91184

add docstring

8b6cdb2

import sort fix

035e73b

fix task_id, dag

d2a45a4

testcase fix

960ee5f

mik-laj added the provider:amazon-aws AWS/Amazon - related issues label Oct 31, 2019

feluelle reviewed Nov 19, 2019

View reviewed changes

airflow/contrib/operators/s3_to_snowflake_operator.py Outdated Show resolved Hide resolved

airflow/contrib/operators/s3_to_snowflake_operator.py Outdated Show resolved Hide resolved

tests/contrib/operators/test_s3_to_snowflake_operator.py Outdated Show resolved Hide resolved

shaikshakeel added 8 commits November 20, 2019 15:45

typo fixes

0d33d9b

add with columns testcase

43660e4

run test cases again

e6ef43d

run test cases

48a414e

run test cases

f4220bc

run testcases

2c04027

Merge branch 'master' of github.com:shaikshakeel/airflow into s3_to_s…

d864f32

…nowflake_operator # Conflicts: # docs/operators-and-hooks-ref.rst

run testcases

685b8fe

shaikshakeel requested a review from feluelle November 20, 2019 16:24

feluelle requested changes Nov 24, 2019

View reviewed changes

shaikshakeel requested a review from feluelle December 2, 2019 11:06

shaikshakeel added 11 commits January 11, 2020 15:34

pylint fix

596657b

ignore snowflake hook in pylint

49bf935

run testcases

06e35d7

run test cases again

225e9d1

run test cases

5d351d4

run test cases

a249b68

run testcases

c7e1ae7

Merge branch 'master' of github.com:shaikshakeel/airflow

3818fda

Merge branch 'master' of github.com:shaikshakeel/airflow into s3_to_s…

38a774a

…nowflake_operator # Conflicts: # scripts/ci/pylint_todo.txt

run test cases

b2a57e3

run testcaess

ab29fb3

shaikshakeel requested a review from feluelle January 13, 2020 10:41

feluelle reviewed Jan 13, 2020

View reviewed changes

airflow/providers/snowflake/hooks/snowflake.py Outdated Show resolved Hide resolved

airflow/providers/snowflake/operators/snowflake.py Show resolved Hide resolved

docs/operators-and-hooks-ref.rst Outdated Show resolved Hide resolved

shaikshakeel and others added 5 commits January 14, 2020 12:41

Update airflow/providers/snowflake/operators/snowflake.py

3aab65c

Co-Authored-By: Felix Uellendall <feluelle@users.noreply.github.com>

review changes

bbe2f61

fix no name in module pylint issue

c1f2ef2

Merge branch 'master' of https://github.com/apache/airflow

e57e2ca

Merge branch 'master' of github.com:shaikshakeel/airflow into s3_to_s…

e4721b4

…nowflake_operator

feluelle reviewed Jan 14, 2020

View reviewed changes

airflow/providers/snowflake/operators/snowflake.py Outdated Show resolved Hide resolved

Update airflow/providers/snowflake/operators/snowflake.py

26ac0b0

Co-Authored-By: Felix Uellendall <feluelle@users.noreply.github.com>

shaikshakeel requested a review from feluelle January 16, 2020 09:09

changes

ce62aa1

feluelle approved these changes Jan 17, 2020

View reviewed changes

feluelle requested review from mik-laj, potiuk and BasPH January 17, 2020 20:30

feluelle merged commit 17af3be into apache:master Jan 21, 2020

feluelle added the AIP-21 label Jan 21, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AIRFLOW-5816] S3 to snowflake operator #6469

[AIRFLOW-5816] S3 to snowflake operator #6469

shaikshakeel commented Oct 30, 2019 •

edited

Loading

codecov-io commented Oct 30, 2019 •

edited

Loading

OmerJog commented Nov 5, 2019

shaikshakeel commented Nov 5, 2019

OmerJog commented Nov 12, 2019

feluelle left a comment

shaikshakeel commented Nov 25, 2019 •

edited

Loading

OmerJog commented Nov 25, 2019

shaikshakeel commented Dec 2, 2019

feluelle commented Dec 2, 2019

feluelle commented Dec 2, 2019

shaikshakeel commented Dec 2, 2019

potiuk commented Dec 9, 2019

[AIRFLOW-5816] S3 to snowflake operator #6469

[AIRFLOW-5816] S3 to snowflake operator #6469

Conversation

shaikshakeel commented Oct 30, 2019 • edited Loading

Jira

Description

Tests

Commits

Documentation

codecov-io commented Oct 30, 2019 • edited Loading

Codecov Report

OmerJog commented Nov 5, 2019

shaikshakeel commented Nov 5, 2019

OmerJog commented Nov 12, 2019

feluelle left a comment

Choose a reason for hiding this comment

shaikshakeel commented Nov 25, 2019 • edited Loading

OmerJog commented Nov 25, 2019

shaikshakeel commented Dec 2, 2019

feluelle commented Dec 2, 2019

feluelle commented Dec 2, 2019

shaikshakeel commented Dec 2, 2019

potiuk commented Dec 9, 2019

shaikshakeel commented Oct 30, 2019 •

edited

Loading

codecov-io commented Oct 30, 2019 •

edited

Loading

shaikshakeel commented Nov 25, 2019 •

edited

Loading