Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Load/export file from Azure blob storage using non-native path #1542

Merged
merged 8 commits into from
Jan 10, 2023

Conversation

tatiana
Copy link
Collaborator

@tatiana tatiana commented Jan 6, 2023

Closes: #1447

Bonus:

  • Improve and standardise export_to_file tests related to remote file locations so we have a consistent way of testing them (all file locations x all remote object stores)
  • Fix the GCS and WASB location exists method

Pending

  • Azure + Databricks support (separate ticket)

Co-authored-by: Luan Moreno luan.moreno@owshq.com
Co-authored-by: Mateus Oliveira mateus.oliveira@owshq.com

@codecov
Copy link

codecov bot commented Jan 6, 2023

Codecov Report

Base: 94.00% // Head: 97.59% // Increases project coverage by +3.58% 🎉

Coverage data is based on head (80c4f46) compared to base (a150cbf).
Patch has no changes to coverable lines.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1542      +/-   ##
==========================================
+ Coverage   94.00%   97.59%   +3.58%     
==========================================
  Files          89       22      -67     
  Lines        4372      789    -3583     
  Branches      432        0     -432     
==========================================
- Hits         4110      770    -3340     
+ Misses        178       19     -159     
+ Partials       84        0      -84     
Impacted Files Coverage Δ
python-sdk/src/astro/files/base.py
python-sdk/src/astro/files/locations/azure/wasb.py
python-sdk/src/astro/files/locations/base.py
python-sdk/src/astro/files/locations/google/gcs.py
python-sdk/src/astro/utils/typing_compat.py
python-sdk/src/astro/sql/table.py
python-sdk/src/astro/databases/base.py
...ricks/load_file/load_file_python_code_generator.py
python-sdk/src/astro/databases/snowflake.py
python-sdk/src/astro/databases/sqlite.py
... and 57 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@tatiana tatiana changed the title Add test case to load file to all databases from Azure blob storage Load file to supported databases from Azure blob storage using non-native path Jan 6, 2023
@tatiana tatiana changed the title Load file to supported databases from Azure blob storage using non-native path Load file from Azure blob storage using non-native path Jan 6, 2023
@tatiana tatiana force-pushed the python-sdk/1447 branch 3 times, most recently from 3daee33 to 04ec544 Compare January 7, 2023 07:32
@tatiana tatiana changed the title Load file from Azure blob storage using non-native path Load/export file from Azure blob storage using non-native path Jan 7, 2023
@tatiana tatiana force-pushed the python-sdk/1447 branch 6 times, most recently from f4c0564 to bc0603f Compare January 9, 2023 10:05
@tatiana tatiana marked this pull request as ready for review January 9, 2023 10:06
Copy link
Contributor

@pankajastro pankajastro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. But look like CI has not triggered on the last commit.

python-sdk/noxfile.py Show resolved Hide resolved
tatiana and others added 4 commits January 9, 2023 14:57
Closes: #1447

Co-authored-by: Luan Moreno <luan.moreno@owshq.com>
Co-authored-by: Mateus Oliveira <mateus.oliveira@owshq.com>
@tatiana
Copy link
Collaborator Author

tatiana commented Jan 9, 2023

@pankajastro thanks! There is something very odd going on with our Github actions; it is triggering the jobs but sometimes it's not deploying here 🤔
https://github.com/astronomer/astro-sdk/actions/runs/3874982912

@pankajastro
Copy link
Contributor

@pankajastro thanks! There is something very odd going on with our Github actions; it is triggering the jobs but sometimes it's not deploying here 🤔
https://github.com/astronomer/astro-sdk/actions/runs/3874982912

I did a force push then it showing up
Screenshot 2023-01-09 at 9 44 59 PM

@tatiana tatiana merged commit ba25f0c into main Jan 10, 2023
@tatiana tatiana deleted the python-sdk/1447 branch January 10, 2023 02:00
utkarsharma2 pushed a commit that referenced this pull request Jan 17, 2023
Closes: #1447

Bonus:
- Improve and standardise `export_to_file` tests related to remote file
locations so we have a consistent way of testing them (all file
locations x all remote object stores)
- Fix the GCS and WASB location `exists` method

Pending
- Azure + Databricks support (separate ticket)

Co-authored-by: Luan Moreno <luan.moreno@owshq.com>
Co-authored-by: Mateus Oliveira <mateus.oliveira@owshq.com>

Co-authored-by: Luan Moreno <luan.moreno@owshq.com>
Co-authored-by: Mateus Oliveira <mateus.oliveira@owshq.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow load and export from Azure Blob storage (WASB) using non-native path
2 participants