rfc: Idempotent Copy

[lichuang]

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

docs: add Idempotent-Copy rfc

Fixes #issue

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated
databend	✅ Ready (Inspect)	Visit Preview	Sep 10, 2022 at 1:19AM (UTC)

[drmingdrmer]

Crystal clear explanation.

One of my question is, in this sentence:
copy the stage file and up-insert into the table stage file meta
These are two RPC operations to databend-meta, right?

BTW, the words in this font is not very easy to read, I'm afraid...

[BohuTANG]

Clear 👍
One question:
does the copy file meta in metaservice with the SQL digest?

COPY into t1 select c1 from @stage;

is not same as:

COPY into t1 select parse_json(c2) from @stage;

[BohuTANG]

Another question:
If we truncate table and COPY from a stage/location again, all the files will be skipped.

[lichuang]

Crystal clear explanation.

One of my question is, in this sentence: copy the stage file and up-insert into the table stage file meta These are two RPC operations to databend-meta, right?

BTW, the words in this font is not very easy to read, I'm afraid...

copy the stage file means copy the stage file into table, it is not an operation on meta service, but like inserting data into a table, so it happen on object store such as s3 in databend.

up-insert into the table stage file meta means up-insert the stage file meta into meta service, so the next time can use these meta data to compare with original stage file meta, and ignore the stage files which has not been modified since last copied.

[lichuang]

Another question:
If we truncate table and COPY from a stage/location again, all the files will be skipped.

emm, yes, this situation has not been considered before...

[lichuang]

Clear 👍 One question: does the copy file meta in metaservice with the SQL digest?

COPY into t1 select c1 from @stage;

is not same as:

COPY into t1 select parse_json(c2) from @stage;

no, the key of table stage copy stage file meta combined with (tenant,database,table,file_name), which has been analyzed by the SQL parser.

[Xuanwo]

StageFile is an internal data type we don't want to persist.

Can we just construct the key the specific way instead of storing StageFile in the meta?

For example:

{prefix}/{tenant}/{db_name}/{table_name}/{file_name}/{content_length}/{etag}/{last_modified}

We just need to check whether this key exists or not without reading its content.

[lichuang]

StageFile is an internal data type we don't want to persist.

Can we just construct the key the specific way instead of storing StageFile in the meta?

For example:

{prefix}/{tenant}/{db_name}/{table_name}/{file_name}/{content_length}/{etag}/{last_modified}

We just need to check whether this key exists or not without reading its content.

since meta now has no mechanism to recycle the expire data, so if the key is {prefix}/{tenant}/{db_name}/{table_name}/{file_name}/{content_length}/{etag}/{last_modified} will make more than one key of a file, i prefer {prefix}/{tenant}/{db_name}/{table_name}/{file_name} -> {content_length}/{etag}/{last_modified} instead.

[Xuanwo]

In an edge case, what will happen if users copy from different stages:

copy from @gcs_stage into mytable;
copy from @aws_stage into mytable;

[lichuang]

In an edge case, what will happen if users copy from different stages:

copy from @gcs_stage into mytable;
copy from @aws_stage into mytable;

it depends on list_files in interpreter_copy_v2.rs, list_files generate all the stage files name of copy.
if two stages has the same file name, it will not be copy again if meta matched.

[Xuanwo]

Vercel built failed: