COPY-3586: Support COPY from external location(S3) [PATCH-1] #4170

BohuTANG · 2022-02-16T07:31:39Z

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

Summary about this PR:

STAGE/COPY parser to plan
Read files from external stage
Write data to table
Add source(CSV/Parquet) builder for easy to use

Remove:

streams/source/source_values not used anymore
stage(will backport under the new design in another PR)

Syntax:

COPY INTO [<database>.]<table_name>
     FROM { externalLocation }
[ FILE_FORMAT = ( TYPE = { CSV | JSON | AVRO | ORC | PARQUET | XML } [ formatTypeOptions ] } ) ]
[ VALIDATION_MODE = RETURN_<n>_ROWS | RETURN_ERRORS ]
[ copyOptions ]

Where

externalLocation (for Amazon S3) ::=
  's3://<bucket>[/<path>]'
  [ { CREDENTIALS = ( {  { AWS_KEY_ID = '<string>' AWS_SECRET_KEY = '<string>' } } ) } ]
  [ ENCRYPTION = (  [ MASTER_KEY = '<string>' ]

formatTypeOptions ::=
-- If FILE_FORMAT = ( TYPE = CSV ... )
     RECORD_DELIMITER = '<character>' 
     FIELD_DELIMITER = '<character>' 
     SKIP_HEADER = <integer>

copyOptions ::=
     ON_ERROR = { CONTINUE | SKIP_FILE | SKIP_FILE_<num>| ABORT_STATEMENT }
     SIZE_LIMIT = <num>

For example:

Create table: ontime/create_table.sql
copy into ontime from 's3://repo.databend.rs/dataset/stateful/ontime.csv' FILE_FORMAT = (type = 'CSV' field_delimiter = ','  record_delimiter = '\n' skip_header = 1)

Changelog

New Feature

Related Issues

Fixes #3586

Test Plan

Unit Tests

Stateless Tests

vercel · 2022-02-16T07:31:43Z

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: https://vercel.com/databend/databend/4uPkvAjbfdYGD3qzxeUgc9aikWBJ
✅ Preview: https://databend-git-fork-bohutang-dev-copy-external-3586-databend.vercel.app

[Deployment for be63b03 canceled]

mergify · 2022-02-16T07:32:12Z

Thanks for the contribution!
I have applied any labels matching special text in your PR Changelog.

Please review the labels and make any necessary changes.

xiaozongyang · 2022-02-23T08:30:44Z

query/src/interpreters/interpreter_copy.rs

+        let table = self
+            .ctx
+            .get_table(&self.plan.db_name, &self.plan.tbl_name)
+            .await?;


Maybe we can verify db_name and tbl_name before accessing file in s3? If wrong db_name/tbl_name is given we can fail.

By the way, I'm fresh on this project -- databend, if some naive comments posted, I'm supposed to offend you guys. Be free to give correct replies.

This PR is under Draft status, database and table privileges check hasn't finished yet, will add them soon.
Thank you.

Do you have some recent deadline about this feature? If not, I'd like to write some code for it.

For COPY INTO TABLE FROM s3 external location feature, I will finish it this week in this PR :)
Well, would like to recommend some issues here:
https://github.com/datafuselabs/databend/issues?q=is%3Aissue+is%3Aopen+label%3A%22C-good+first+issue%22

BohuTANG · 2022-02-23T11:36:05Z

This PR should merge with #4203, but we can review it now.

Xuanwo · 2022-02-24T01:15:45Z

Ping @BohuTANG, please give me your fork's write permission to resolve the conflicts. 😆

BohuTANG · 2022-02-24T03:05:50Z

Wait for OpenDAL:
backend/s3: Anonymous access not supported
backend/s3: Remove the requirement for region

BohuTANG · 2022-02-24T03:09:02Z

Hello,

This feature is not available at the moment(only works with minio), it needs to wait for OpenDAL to finish:
apache/opendal#54
apache/opendal#56

Since there are too many files involved, I will merge it first as the PATCH-1.
I will make it works with AWS S3 in another PR as PATCH-2 after OpenDAL#54 and OpenDAL#56 finished.

Xuanwo · 2022-02-24T03:07:40Z

common/streams/tests/it/sources/source_csv.rs

+            ]);
+
+            let local = Operator::new(
+                fs::Backend::build()


Self-reminder: Implement apache/opendal#30 so that users can test easier.

Xuanwo · 2022-02-24T03:13:31Z

query/src/interpreters/interpreter_copy.rs

+            StageType::External => {
+                match plan.stage_info.file_format_options.format {
+                    // CSV.
+                    StageFileFormatType::Csv => {


It seems we support parquet too?

Yes, I will add the parquet support after the csv works all well :D

COPY-3586: Support COPY from stage and external location(S3)

ad4212d

databend-bot added the need-review label Feb 16, 2022

vercel bot temporarily deployed to Preview February 16, 2022 07:31 Inactive

mergify bot added the pr-feature this PR introduces a new feature to the codebase label Feb 16, 2022

Add Stage Plan

fca9efa

vercel bot temporarily deployed to Preview February 17, 2022 04:35 Inactive

Add new UserStageInfo

e207d18

vercel bot temporarily deployed to Preview February 17, 2022 10:48 Inactive

Revoke stage create/drop

5fe1026

vercel bot temporarily deployed to Preview February 17, 2022 11:48 Inactive

Remove stage tests

ac2aaf0

vercel bot temporarily deployed to Preview February 17, 2022 11:55 Inactive

Add copy statement analyze test

072c62b

vercel bot temporarily deployed to Preview February 18, 2022 06:51 Inactive

Try add interpreter copy

a7ca76c

Xuanwo mentioned this pull request Feb 18, 2022

LOAD_FILE() | Load the named file #3059

Closed

Try to add interpreter copy

aef4712

vercel bot temporarily deployed to Preview February 20, 2022 03:08 Inactive

Merge from main

480522e

vercel bot deployed to Preview February 20, 2022 05:33 View deployment

BohuTANG added 2 commits February 20, 2022 13:36

Remove uriparse dep

cab689b

field/record_delimitor -> field/record_delimiter

391bdf9

vercel bot deployed to Preview February 20, 2022 08:55 View deployment

Add file format option to user stage info

b86180a

vercel bot temporarily deployed to Preview February 20, 2022 10:10 Inactive

Add interpreter copy

f086d19

vercel bot temporarily deployed to Preview February 20, 2022 10:47 Inactive

Fix statement copy unit test

f874a03

vercel bot temporarily deployed to Preview February 23, 2022 05:50 Inactive

Add endpoint to get opendal from the config

a1e97a9

vercel bot temporarily deployed to Preview February 23, 2022 06:51 Inactive

Merge branch 'main' into dev-copy-external-3586

d53ffe6

vercel bot deployed to Preview February 23, 2022 06:52 View deployment

Add ontime 200lines test dataset

4287e96

vercel bot temporarily deployed to Preview February 23, 2022 07:14 Inactive

Add credential to copy statful test

6082071

vercel bot temporarily deployed to Preview February 23, 2022 07:50 Inactive

xiaozongyang reviewed Feb 23, 2022

View reviewed changes

Fix configs_table

ee78367

vercel bot temporarily deployed to Preview February 23, 2022 09:07 Inactive

Fix statement copy unit test and refine the copy plan debug format

902f06d

vercel bot temporarily deployed to Preview February 23, 2022 10:20 Inactive

Add docs/user/01-sql-statement/03-dml/dml-copy.md

dec7efd

vercel bot deployed to Preview February 23, 2022 10:38 View deployment

BohuTANG marked this pull request as ready for review February 23, 2022 11:36

BohuTANG requested review from sundy-li and Xuanwo February 23, 2022 13:03

Merge with main

be63b03

vercel bot temporarily deployed to Preview February 24, 2022 02:19 Inactive

sundy-li approved these changes Feb 24, 2022

View reviewed changes

BohuTANG changed the title ~~COPY-3586: Support COPY from external location(S3)~~ COPY-3586: Support COPY from external location(S3) [PATCH-1] Feb 24, 2022

BohuTANG merged commit 20648c3 into datafuselabs:main Feb 24, 2022

BohuTANG mentioned this pull request Feb 24, 2022

Copy INTO <table> from external location #3586

Closed

7 tasks

Xuanwo reviewed Feb 24, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

COPY-3586: Support COPY from external location(S3) [PATCH-1] #4170

COPY-3586: Support COPY from external location(S3) [PATCH-1] #4170

BohuTANG commented Feb 16, 2022 •

edited

vercel bot commented Feb 16, 2022 •

edited

mergify bot commented Feb 16, 2022

xiaozongyang Feb 23, 2022

xiaozongyang Feb 23, 2022

BohuTANG Feb 23, 2022 •

edited

xiaozongyang Feb 23, 2022

BohuTANG Feb 23, 2022

BohuTANG commented Feb 23, 2022

Xuanwo commented Feb 24, 2022

BohuTANG commented Feb 24, 2022

BohuTANG commented Feb 24, 2022 •

edited

Xuanwo Feb 24, 2022

Xuanwo Feb 24, 2022

BohuTANG Feb 24, 2022

COPY-3586: Support COPY from external location(S3) [PATCH-1] #4170

COPY-3586: Support COPY from external location(S3) [PATCH-1] #4170

Conversation

BohuTANG commented Feb 16, 2022 • edited

Summary

Changelog

Related Issues

Test Plan

vercel bot commented Feb 16, 2022 • edited

mergify bot commented Feb 16, 2022

xiaozongyang Feb 23, 2022

Choose a reason for hiding this comment

xiaozongyang Feb 23, 2022

Choose a reason for hiding this comment

BohuTANG Feb 23, 2022 • edited

Choose a reason for hiding this comment

xiaozongyang Feb 23, 2022

Choose a reason for hiding this comment

BohuTANG Feb 23, 2022

Choose a reason for hiding this comment

BohuTANG commented Feb 23, 2022

Xuanwo commented Feb 24, 2022

BohuTANG commented Feb 24, 2022

BohuTANG commented Feb 24, 2022 • edited

Xuanwo Feb 24, 2022

Choose a reason for hiding this comment

Xuanwo Feb 24, 2022

Choose a reason for hiding this comment

BohuTANG Feb 24, 2022

Choose a reason for hiding this comment

BohuTANG commented Feb 16, 2022 •

edited

vercel bot commented Feb 16, 2022 •

edited

BohuTANG Feb 23, 2022 •

edited

BohuTANG commented Feb 24, 2022 •

edited