-
Asking here as I do not see a link to a community discourse/matrix/mailing list/IRC on duckdb.org. Please correct me if I am wrong. Lotsa CSVs questionI have 200 CSV files each with about on million rows. So about 200 million rows. Naturally I don't want to insert these one by one, but the only documentation in the python api seems to give this recommendation for multiple inserts: So should I open each CSV one by one, and use this technique for each batch of say 1000 rows? What are my other options for batch uploading tons of CSVs? I see in the importing CSV section that you can batch import a CSV (I assume these instructions refer to the CLI version of duckdb, because this is not specified), but how do you batch import 200 CSVs without naming each one individually (they all have identical schema). |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 2 replies
-
Both the
All of the instructions there are in SQL, which can be used from every client. |
Beta Was this translation helpful? Give feedback.
-
We do have a discord channel! |
Beta Was this translation helpful? Give feedback.
Both the
COPY
statement and theread_csv_auto
function support wildcards, so you can do e.g.COPY tbl FROM 'test*.csv' (FORMAT CSV, ...);
All of the instructions there are in SQL, which can be used from every client.