Parquet filter pushdown

[filimonov]

Limit reads from parquet file, when filters exist: similar https://drill.apache.org/docs/parquet-filter-pushdown/

[filimonov]

That was implemented for Hive (only) some time ago. Same is needed for file(..) and s3(...)

#34631

[danthegoodman1]

Reviving this because it's super useful, duckdb has this and that makes an insane difference in more selective queries on larger files

[minguyen9988]

second it, on select with where query duckdb vastly outperform Clickhouse (10+ times ).

[danthegoodman1]

That was implemented for Hive (only) some time ago. Same is needed for file(..) and s3(...)

#34631

Don't forget url() (and *cluster versions!)

[danthegoodman1]

second it, on select with where query duckdb vastly outperform Clickhouse (10+ times ).

For reference, you can see clickhouse is faster until the queries have any sort of selectivity in such a way that the minxmax index of the file can be used. In my experience, I am always filtering so I am forced to use DuckDB for the performance gains of parquet files.

[danthegoodman1]

Don’t forget to update clickbench!