Canonical way of batch uploading a whole lot of CSVs into DuckDB from Python

[vegabook]

Asking here as I do not see a link to a community discourse/matrix/mailing list/IRC on duckdb.org. Please correct me if I am wrong.

Lotsa CSVs question

I have 200 CSV files each with about on million rows. So about 200 million rows. Naturally I don't want to insert these one by one, but the only documentation in the python api seems to give this recommendation for multiple inserts:

So should I open each CSV one by one, and use this technique for each batch of say 1000 rows? What are my other options for batch uploading tons of CSVs? I see in the importing CSV section that you can batch import a CSV (I assume these instructions refer to the CLI version of duckdb, because this is not specified), but how do you batch import 200 CSVs without naming each one individually (they all have identical schema).

[Mytherin]

Both the COPY statement and the read_csv_auto function support wildcards, so you can do e.g.

COPY tbl FROM 'test*.csv' (FORMAT CSV, ...);

I assume these instructions refer to the CLI version of duckdb, because this is not specified

All of the instructions there are in SQL, which can be used from every client.

[hannes]

We do have a discord channel!

[vegabook]

We do have a discord channel!

Okay so did I not find it on the site? How do I join it?

[rsund]

https://discord.gg/tcvwpjfnZx