Support copy command to load csv/parquet or s3 file into table

[sundy-li]

Summary

Description for this feature.

support copy command to load csv file into table.

COPY table_name FROM '/path/to/test.csv'

task list:

• support copy stage csv/parquet file into table Support Copy command from stage location into table #2878
• support async read stream to parse formats Async datastream #2786
• [query] http streaming load #3082

Reference:
https://www.postgresql.org/docs/13/sql-copy.html
https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html
https://duckdb.org/docs/sql/statements/copy

[ZhiHanZ]

I think it should be a query semantic in http handler?

[sundy-li]

I think it should be a query semantic in http handler?

But we need to upload localfile, how could http handler handle this?

[BohuTANG]

I think it should be a query semantic in http handler?

But we need to upload localfile, how could http handler handle this?

1. http handler provides a put file(like xx_yy.csv) command to a path as tmp_dir which configed in the query -- this upload a file to the internal stage
2. query supports the COPY command: COPY INTO table_name FROM 'xx_yy.csv' -- query read the file from the internal stage

https://docs.snowflake.com/en/sql-reference/sql/put.html
https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html#loading-files-from-an-internal-stage

[flaneur2020]

I think it should be a query semantic in http handler?

But we need to upload localfile, how could http handler handle this?

1. http handler provides a put file(like xx_yy.csv) command to a path as tmp_dir which configed in the query -- this upload a file to the internal stage
2. query supports the COPY command: COPY INTO table_name FROM 'xx_yy.csv' -- query read the file from the internal stage

https://docs.snowflake.com/en/sql-reference/sql/put.html https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html#loading-files-from-an-internal-stage

the csv file have to located on the bend-query instance's local filesystem? 🤔

[flaneur2020]

how about copy file from a s3 or hdfs path? like:

COPY INTO table_name FROM 's3://xx_yy.csv?s3_token=xxxx'

[sundy-li]

how about copy file from a s3 or hdfs path? like:

COPY INTO table_name FROM 's3://xx_yy.csv?s3_token=xxxx'

Yes, we can judge the protocol like "s3", "http", "hdfs", "file". I'll implement a base version to support local filesystem.

[BohuTANG]

Are there any more tasks for this issue?