-
-
Notifications
You must be signed in to change notification settings - Fork 7
Open
Description
Hello
We want to use chdb for our ETL tasks.
But all rows of response is buffered into string, this is not suitable when you need to read a milions rows.
We, and I think many others, need a streaming row-by-row response.
In my opinion, a good interface would be a new queryStream method that returns an asynchronous generator.
This would provide good performance and two-way control.
Each reading iteration is called asynchronously.
You can also load data asynchronously. Here's an example:
for await (const row of queryStream(`SELECT * FROM file('data.parquet', Parquet)`, 'JSONEachRow'))
// processing by row
);Is it possible to do this?
DuckDB has similar functionality, the absence of this functionality makes it impossible to use chdb to obtain many rows.
Thanks.
Metadata
Metadata
Assignees
Labels
No labels