Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Snowflake - azure native path #1675

Merged
merged 14 commits into from
Feb 3, 2023
Merged

Snowflake - azure native path #1675

merged 14 commits into from
Feb 3, 2023

Conversation

utkarsharma2
Copy link
Collaborator

@utkarsharma2 utkarsharma2 commented Jan 25, 2023

Description

What is the current behavior?

Right now if we have to load data from Azure to snowflake we have to involve the worker node, which is time taking process.

closes: #1486
closes: #1448

What is the new behavior?

Add Snowflake - Azure native path to load data directly from Azure to Snowflake, without involving a worker node.

Does this introduce a breaking change?

Nope

Checklist

  • Created tests which fail without the change (if possible)
  • Extended the README / documentation, if necessary

@codecov
Copy link

codecov bot commented Jan 25, 2023

Codecov Report

Base: 97.72% // Head: 93.20% // Decreases project coverage by -4.53% ⚠️

Coverage data is based on head (2c11011) compared to base (b919eec).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1675      +/-   ##
==========================================
- Coverage   97.72%   93.20%   -4.53%     
==========================================
  Files          21       90      +69     
  Lines         835     4736    +3901     
  Branches        0      470     +470     
==========================================
+ Hits          816     4414    +3598     
- Misses         19      232     +213     
- Partials        0       90      +90     
Impacted Files Coverage Δ
python-sdk/src/astro/constants.py 94.23% <100.00%> (ø)
python-sdk/src/astro/databases/snowflake.py 86.26% <100.00%> (ø)
python-sdk/src/astro/files/locations/__init__.py 100.00% <100.00%> (ø)
python-sdk/src/astro/files/locations/azure/wasb.py 98.43% <100.00%> (ø)
python-sdk/src/astro/files/locations/base.py 79.09% <100.00%> (ø)
python-sdk/src/astro/options.py 97.14% <100.00%> (ø)
...ro/databases/databricks/load_file/load_file_job.py 96.77% <0.00%> (ø)
python-sdk/src/astro/databases/__init__.py 100.00% <0.00%> (ø)
python-sdk/src/astro/utils/path.py 100.00% <0.00%> (ø)
...tro/sql/operators/data_validations/check_column.py 86.30% <0.00%> (ø)
... and 65 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@utkarsharma2 utkarsharma2 marked this pull request as ready for review January 30, 2023 09:53
Copy link
Contributor

@pankajastro pankajastro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@utkarsharma2 utkarsharma2 merged commit e5f85e9 into main Feb 3, 2023
@utkarsharma2 utkarsharma2 deleted the snowflake_azure branch February 3, 2023 10:58
utkarsharma2 added a commit that referenced this pull request Feb 8, 2023
# Description
## What is the current behavior?
Right now if we have to load data from Azure to snowflake we have to
involve the worker node, which is time taking process.

closes: #1486 
closes: #1448

## What is the new behavior?
Add Snowflake - Azure native path to load data directly from Azure to
Snowflake, without involving a worker node.


## Does this introduce a breaking change?
Nope


### Checklist
- [ ] Created tests which fail without the change (if possible)
- [ ] Extended the README / documentation, if necessary

---------

Co-authored-by: Felix Uellendall <feluelle@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants