Skip to content

Streaming of query result #31

@uasan

Description

@uasan

Hello
We want to use chdb for our ETL tasks.
But all rows of response is buffered into string, this is not suitable when you need to read a milions rows.

We, and I think many others, need a streaming row-by-row response.

In my opinion, a good interface would be a new queryStream method that returns an asynchronous generator.
This would provide good performance and two-way control.
Each reading iteration is called asynchronously.

You can also load data asynchronously. Here's an example:

for await (const row of queryStream(`SELECT * FROM file('data.parquet', Parquet)`, 'JSONEachRow'))
 // processing by row
);

Is it possible to do this?
DuckDB has similar functionality, the absence of this functionality makes it impossible to use chdb to obtain many rows.

Thanks.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions