Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support loading datasets of 10GB in Snowflake in less than 5 min #430

Closed
4 tasks
tatiana opened this issue Jun 6, 2022 · 1 comment · Fixed by #544
Closed
4 tasks

Support loading datasets of 10GB in Snowflake in less than 5 min #430

tatiana opened this issue Jun 6, 2022 · 1 comment · Fixed by #544
Assignees
@tatiana @sunank200
Labels
improvement Enhancement or improvement in an existing feature
Milestone
1.0.0

Comments

@tatiana
Copy link
Collaborator

tatiana commented Jun 6, 2022 •
edited by sunank200
Loading

Dependencies

Acceptance criteria

  • Make changes to how we load data into Snowflake.
  • Re-run the benchmark and identify the performance improvements (?)
  • The changes must work for all file types supported (CSV, JSON, NDJSON, Parquet)
  • The changes made would also work on S3 and GCS
@kaxil kaxil added this to the 1.0.0 milestone Jun 6, 2022
@tatiana tatiana changed the title Support loading datasets of 10GB in Snowflake in less than 2 min Support loading datasets of 10GB in Snowflake in less than 5 min Jun 7, 2022
@tatiana tatiana added the improvement Enhancement or improvement in an existing feature label Jun 7, 2022
@kaxil kaxil assigned kaxil and sunank200 and unassigned tatiana and kaxil Jun 22, 2022
@sunank200 sunank200 mentioned this issue Jun 23, 2022
Closed
2 tasks
@tatiana tatiana self-assigned this Jul 13, 2022
tatiana added a commit that referenced this issue Jul 14, 2022
@tatiana
Refactor table creation in load_file (#538)
b77e22e 
Refactor how tables are created in BaseDatabase.load_file_to_table

We should prioritise creating the table using the `table.columns` if they are specified by the user and have the dataframe autodetection as a fallback.

Most of the complexity of #487 was the creation of tables, and this step aims to simplify the Snowflake `load_file` optimization.

Relates to: #430, #481, #493, #494
tatiana added a commit that referenced this issue Jul 15, 2022
@tatiana
Add Snowflake load_file optimization
63e83db 
Fix: #430

Co-authored-by: Ankit Chaurasia <ankit.chaurasia@astronomer.io>
@tatiana tatiana mentioned this issue Jul 15, 2022
Merged
tatiana added a commit that referenced this issue Jul 18, 2022
@tatiana
Add Snowflake load_file optimization
3a02d8a 
Fix: #430

Co-authored-by: Ankit Chaurasia <ankit.chaurasia@astronomer.io>
tatiana added a commit that referenced this issue Jul 19, 2022
@tatiana
Add Snowflake load_file optimization
2c3972d 
Fix: #430

Co-authored-by: Ankit Chaurasia <ankit.chaurasia@astronomer.io>
tatiana added a commit that referenced this issue Jul 19, 2022
@tatiana
Add Snowflake load_file optimization
c8284d8 
Fix: #430

Co-authored-by: Ankit Chaurasia <ankit.chaurasia@astronomer.io>
kaxil pushed a commit that referenced this issue Jul 20, 2022
@tatiana @kaxil
Add Snowflake load_file optimization
25598fe 
Fix: #430

Co-authored-by: Ankit Chaurasia <ankit.chaurasia@astronomer.io>
tatiana added a commit that referenced this issue Jul 20, 2022
@tatiana
Add Snowflake load_file optimization
3f1f757 
Fix: #430

Co-authored-by: Ankit Chaurasia <ankit.chaurasia@astronomer.io>
@tatiana tatiana reopened this Jul 20, 2022
@tatiana
Copy link
Collaborator Author

tatiana commented Jul 20, 2022

I believe this was closed by mistake, we haven't merged the changes into master yet..!

sunank200 pushed a commit that referenced this issue Jul 22, 2022
@tatiana @sunank200
Add Snowflake load_file optimization
5382bee 
Fix: #430

Co-authored-by: Ankit Chaurasia <ankit.chaurasia@astronomer.io>
utkarsharma2 pushed a commit that referenced this issue Jul 25, 2022
@tatiana @utkarsharma2
Add Snowflake load_file optimization
d6a274f 
Fix: #430

Co-authored-by: Ankit Chaurasia <ankit.chaurasia@astronomer.io>
tatiana added a commit that referenced this issue Jul 25, 2022
@tatiana
Add Snowflake load_file optimization
da597a2 
Fix: #430

Co-authored-by: Ankit Chaurasia <ankit.chaurasia@astronomer.io>
tatiana added a commit that referenced this issue Jul 25, 2022
@tatiana
Add Snowflake load_file optimization
ecedb2b 
Fix: #430

Co-authored-by: Ankit Chaurasia <ankit.chaurasia@astronomer.io>
tatiana added a commit that referenced this issue Jul 26, 2022
@tatiana
Optimize Snowflake load_file using native COPY INTO (#544)
461e63e 
Fix: #430 

Reduce the time to load to Snowflake by 20% for 5GB datasets (from 24.46 min to 5.49 min). Further details are in the PR results file.

Co-authored-by: Ankit Chaurasia <ankit.chaurasia@astronomer.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tatiana tatiana

@sunank200 sunank200

Labels
improvement Enhancement or improvement in an existing feature
Projects
None yet
Milestone
1.0.0
4 participants
@tatiana @dimberman @sunank200 @kaxil