Parquet Scan using Stream

[ravi27may]

Is there a way to read parquet data from parquet stream using Parquet_Scan function

[Alex-Monahan]

Hello!

Unfortunately Parquet files store their metadata at the end, so there isn't an effective way to stream them. You would have to read in all of the data to understand what you were reading!

Here is a previous issue that discusses it in more depth. #2826

If you would like to share more about your use case, maybe we can find a good workaround for you!

[ravi27may]

Thanks @Alex-Monahan for the quick response.

Am receiving the parquet stream from a different source which in my app i need to consume it as source and extract only little data from that stream. I read with duckdb we can able to use parquet_scan to query .parquet files locally, so i thought it's possible to read the data as stream (as file as read as stream internally)

Also, if there is a way where we can query s3 files directly that also helps. I tried to use duckdb cli (windows 64) but it seems it's not working. Version used 0.3.1

[ravi27may]

Also, i tried to use the examples from this link. None of them are actually working with windows cli
https://github.com/duckdb/duckdb/blob/1fbc35f37e2b5b5e641ae0e0cb145078621c0c7b/test/sql/copy/parquet/test_parquet_remote.test

[Mytherin]

With version 0.3.2 you should be able to read Parquet files from S3 by installing the HTTPFS extension. You can do this by running the following commands:

INSTALL httpfs;
LOAD httpfs;

[ravi27may]

Thanks Mytherin. Do we have any alternative way to read from stream

[Mytherin]

You can use Arrow to read Parquet files from a stream and then read the resulting Arrow stream using DuckDB.