[Q&A] Why is querying Hive partitioning parquets slow?

[xqe2011]

What happens?

Recently, we tried some duckdb-related functions of paradedb instead of using a standalone duckdb instance. When we run a simple SELECT query on paradedb, it's 2-20 times slower than DuckDB.

Profiling method
SELECT duckdb_execute($$SET enable_profiling='query_tree'$$); and watch paradedb logs.

To Reproduce

Query one field : SELECT name FROM public.table1 where code1 = 3261 and code2 = '001' and code3 = '5204' and code4 = '1'
code1 and code2 are partition fields.

1. DuckDB runs on cli 0.0190s
2. DuckDB runs in paradedb 0.0291s
3. ParadeDB 0.513s

Query multi fields: SELECT name, level, detxlen, detylen, downid FROM public.table1 where code1 = 3261 and code2 = '001' and code3 = '5204' and code4 = '1'
code1 and code2 are partition fields.

1. DuckDB runs on cli 0.0413s
2. DuckDB runs in paradedb 0.037s
3. ParadeDB 0.552s

OS:

Ubuntu Server 22.04.3

ParadeDB Version:

paradedb/paradedb:16-v0.11.1

Are you using ParadeDB Docker, Helm, or the extension(s) standalone?

ParadeDB Helm Chart

Full Name:

Liu Qijie

Affiliation:

Dongguan University of Technology

Did you include all relevant data sets for reproducing the issue?

Yes

Did you include the code required to reproduce the issue?

• Yes, I have

Did you include all relevant configurations (e.g., CPU architecture, PostgreSQL version, Linux distribution) to reproduce the issue?

• Yes, I have

[philippemnoel]

Hey! I'm not sure what the cause might be top-of-mind. Could you please open a GitHub issue in pg_analytics and someone will take a look?

[xqe2011]

Thanks, I opened an issue here

[philippemnoel]

Excellent, thank you!