Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Copy INTO <table> from external location #3586

Closed
7 tasks done
Tracked by #4104
BohuTANG opened this issue Dec 22, 2021 · 9 comments · Fixed by #4170
Closed
7 tasks done
Tracked by #4104

Copy INTO <table> from external location #3586

BohuTANG opened this issue Dec 22, 2021 · 9 comments · Fixed by #4170
Assignees
Labels
A-query Area: databend query

Comments

@BohuTANG
Copy link
Member

BohuTANG commented Dec 22, 2021

Summary
Load all files in a s3 bucket and insert into table:

copy into mytable
  from 's3://mybucket/data/files'
  credentials=(aws_key_id='$AWS_ACCESS_KEY_ID' aws_secret_key='$AWS_SECRET_ACCESS_KEY')
  encryption=(master_key = 'eSxX0jzYfIamtnBKOEOwq80Au6NbSgPH5r4BDDwOaO8=')
  file_format = (type = csv field_delimiter = '|' skip_header = 1);

Tasks:

@BohuTANG BohuTANG added A-query Area: databend query good first issue Category: good first issue labels Dec 22, 2021
@BohuTANG
Copy link
Member Author

We need some data accessor apis to help the file reading, maybe we can address it in:
#3575

cc @Xuanwo

@Xuanwo
Copy link
Member

Xuanwo commented Dec 22, 2021

I'm working on #3436 now.

I believe we will have a more elegant API design to support those feature requests.

@GrapeBaBa
Copy link
Contributor

need wait #3436

@BohuTANG
Copy link
Member Author

need wait #3436

That's right!

@GrapeBaBa
Copy link
Contributor

@Xuanwo can this task start right now?

@Xuanwo
Copy link
Member

Xuanwo commented Jan 17, 2022

@GrapeBaBa, sorry for the late, I think we still need to wait for #3874, I expect it can be finished in this week.

@BohuTANG BohuTANG changed the title Copy into table from s3 file Copy INTO <table> from external location Feb 11, 2022
@BohuTANG BohuTANG mentioned this issue Feb 11, 2022
4 tasks
@BohuTANG BohuTANG removed the good first issue Category: good first issue label Feb 11, 2022
@BohuTANG BohuTANG self-assigned this Feb 14, 2022
@BohuTANG
Copy link
Member Author

Plan to experience the new dal design, I take this issue.
Feel free to comments @GrapeBaBa

@BohuTANG
Copy link
Member Author

This feature need openDAL list: apache/opendal#12 cc @Xuanwo

@BohuTANG
Copy link
Member Author

Ready to close, and #4308 rolling to next version(Nightly v0.8)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-query Area: databend query
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants