DuckDB Source Read Parameters

[jnb1967]

DuckDB supports quite a number of parameters for loading the supported sources. Do you currently or plan to have the ability to manually control setting additional parameter options that are not presented in your UI (Basic Tab)? For example, I want to read all CSV files from a folder (wildcard in filename), and have the filename=True option so I can additionally extract data from the filename. Alternatively, allow me execute the read using my own custom SQL?

[krotname]

From the current source, the CSV node does not expose arbitrary DuckDB read_csv_auto() options yet.

The CSV UI currently maps to a fixed set of properties: path, header, delimiter, quote character, encoding, skipped lines, null sentinel, plus schema/date-format handling. The engine then builds read_csv_auto(...) from that fixed list. I do not see a generic “extra reader options” field on src.csv, so filename=true is not currently available from the basic CSV source.

For your specific case, the closest current workaround is to use a DuckDB source with Advanced custom SQL and call DuckDB directly, for example:

SELECT *
FROM read_csv_auto('/path/to/folder/*.csv', filename = true);

If the UI requires a DuckDB database path for that source, point it at a small/empty .duckdb file; the custom query itself does not have to read from duckle_src unless you reference attached tables.

Longer term, I think this would fit well as either:

• a dedicated filename checkbox on CSV/Parquet/JSON file sources, and/or
• an “advanced DuckDB reader options” text/JSON field that is appended to the generated read_csv_auto() call.

That would cover this without forcing the whole node into custom SQL.

[SouravRoy-ETL]

Nice one guys! I will review and implement soon!

[SouravRoy-ETL]

Done in v0.5.0. src.csv now passes extra DuckDB read parameters straight through (readOptions), and there is a filename option too. So the CSV reader params you need go right into the read.

Please download - https://github.com/slothflowlabs/duckle/releases/tag/v0.5.0 and let me know!