rowid filter pushdown

[Maxxen]

This PR adds support for pushing down static and dynamic filters on the rowid column into table scans.

The rowid column is not a real stored column and doesn't have statistics, but since row-ids are sequential within a row group we can always trivially create fake statistics on-the-fly to feed to the table filters. We also create dummy stats and filter for each vector when doing a partial scan of a row-group in order to avoid materializing the row-ids and actually evaluate the filter expression fully when there are no matches.

This opens up new possibilities for query-rewrites and optimizations.

Benchmarks:

Here are some microbenchmarks, all run on TPC-H sf=10

Static Filter Pushdown

This benchmark just does a point-lookup, with or without the filter pushdown optimizer enabled.

CALL dbgen(sf=10);
SELECT * FROM lineitem where rowid = 500;

Results:

Without rowid pushdown:

benchmark/micro/pushdown/rowid_no_pushdown.benchmark
name    run     timing
benchmark/micro/pushdown/rowid_no_pushdown.benchmark    1       0.292741
benchmark/micro/pushdown/rowid_no_pushdown.benchmark    2       0.291047
benchmark/micro/pushdown/rowid_no_pushdown.benchmark    3       0.292847
benchmark/micro/pushdown/rowid_no_pushdown.benchmark    4       0.297395
benchmark/micro/pushdown/rowid_no_pushdown.benchmark    5       0.292097

With rowid pushdown:

benchmark/micro/pushdown/rowid_pushdown.benchmark   
name    run     timing
benchmark/micro/pushdown/rowid_pushdown.benchmark       1       0.000455
benchmark/micro/pushdown/rowid_pushdown.benchmark       2       0.000462
benchmark/micro/pushdown/rowid_pushdown.benchmark       3       0.000586
benchmark/micro/pushdown/rowid_pushdown.benchmark       4       0.000522
benchmark/micro/pushdown/rowid_pushdown.benchmark       5       0.000428

Dynamic Filter Pushdown

Here we're rewriting a Top-N style query to instead produce a dynamic filter on the rowid as a result of the comparison join produced by the IN clause. In the future we may want to rewrite this query automatically to benefit from this optimization when a lot of columns are returned.

-- The "naive" Top-N
SELECT * FROM lineitem ORDER BY l_orderkey DESC LIMIT 5;

-- Is equivalent to this rewritten query:
SELECT * FROM lineitem WHERE rowid IN (SELECT rowid FROM lineitem ORDER BY l_orderkey DESC LIMIT 5);

Results:

Naive Top-N

benchmark/micro/pushdown/rowid_no_pushdown_dynamic.benchmark    1       0.381389
benchmark/micro/pushdown/rowid_no_pushdown_dynamic.benchmark    2       0.372278
benchmark/micro/pushdown/rowid_no_pushdown_dynamic.benchmark    3       0.372439
benchmark/micro/pushdown/rowid_no_pushdown_dynamic.benchmark    4       0.379967
benchmark/micro/pushdown/rowid_no_pushdown_dynamic.benchmark    5       0.372103

Rewritten Top-N with manual row filtering (on this branch, with rowid pushdown):

benchmark/micro/pushdown/rowid_pushdown_dynamic.benchmark   
name    run     timing
benchmark/micro/pushdown/rowid_pushdown_dynamic.benchmark       1       0.101268
benchmark/micro/pushdown/rowid_pushdown_dynamic.benchmark       2       0.101460
benchmark/micro/pushdown/rowid_pushdown_dynamic.benchmark       3       0.101515
benchmark/micro/pushdown/rowid_pushdown_dynamic.benchmark       4       0.104626
benchmark/micro/pushdown/rowid_pushdown_dynamic.benchmark       5       0.101839

Rewritten Top-N with manual row filtering, on main branch (without rowid pushdown):

./build/release/benchmark/benchmark_runner benchmark/micro/pushdown/main_dynamic_rowid_pushdown.benchmark 
name    run     timing
benchmark/micro/pushdown/main_dynamic_rowid_pushdown.benchmark  1       0.414385
benchmark/micro/pushdown/main_dynamic_rowid_pushdown.benchmark  2       0.411004
benchmark/micro/pushdown/main_dynamic_rowid_pushdown.benchmark  3       0.403185
benchmark/micro/pushdown/main_dynamic_rowid_pushdown.benchmark  4       0.405434
benchmark/micro/pushdown/main_dynamic_rowid_pushdown.benchmark  5       0.409918

[Maxxen]

I haven't benchmarked it specifically, but I'd also expect the duckdb_fts extension to benefit from this as AFAIK it stores row-ids in the FTS-index it creates and uses regular joins to do lookups.

[carlopi]

Amazing work!

I think also queries like:

SELECT * FROM lineitem LIMIT 12000 OFFSET 1000000;

once rewritten as

SELECT * FROM lineitem WHERE rowid >=1000000  LIMIT 12000;

are significantly faster with this PR, allowing reliably fast access to arbitrary segments of a table.

---

Or this is great also for sampling (this is with reservoir, where there is biggest overhead, but other sampling strategies on table can possibly also benefit from being seen as: first identify which rowid, then fetch)

SELECT * FROM lineitem USING SAMPLE 5% (reservoir);
--- Run Time (s): real 9.636 user 11.249101 sys 0.411607

Two steps example in SQL, first create selection then use rowid pushdown

CREATE OR REPLACE TABLE selection AS (SELECT * FROM range(1,(SELECT count(*) FROM lineitem)) USING SAMPLE 5% (reservoir ));
--- Run Time (s): real 1.313 user 1.289073 sys 0.233425
SELECT * FROM lineitem WHERE rowid IN (FROM selection);
--- Run Time (s): real 0.653 user 4.730228 sys 0.077837

Also note that the bottleneck becomes the selection table creation, and significantly faster way to populate the selection vector. (for sure in C++, possibly mocked in SQL like:)

CREATE OR REPLACE TABLE selection_alternative AS (SELECT DISTINCT (random()*(SELECT count(*) FROM lineitem))::INT FROM range (1, (SELECT count(*) FROM lineitem)//18)) LIMIT (SELECT count(*) FROM lineitem)//20;
--- Run Time (s): real 0.200 user 0.309626 sys 0.182240

[Mytherin]

Thanks! Looks great - some minor comments:

[Maxxen]

@Mytherin Seems like a spurious python download failure, could you restart the failing job for the extension CI?

[Mytherin]

Thanks for the PR! LGTM - could we perhaps add a few tests where we use the rowid pushdown as an optimization, e.g.:

SELECT * FROM lineitem WHERE rowid IN (SELECT rowid FROM lineitem ORDER BY l_orderkey DESC LIMIT 5);
SELECT * FROM lineitem WHERE rowid IN (SELECT rowid FROM lineitem WHERE l_orderkey * l_suppkey = 5998800238368);

[arjenpdevries]

I have a question about this very useful optimisation.

I think select * from ... where rowid in (select rowid from ...) can also be seen as positional join.

How do these two ways of processing these "batch array lookup" requests compare?

[Mytherin]

rowid can have gaps if there are deletes, e.g.:

D create table tbl(i int);
D insert into tbl values (1), (2), (3);
D delete from tbl where i=2;
D select i, rowid from tbl;
┌───────┬───────┐
│   i   │ rowid │
│ int32 │ int64 │
├───────┼───────┤
│     1 │     0 │
│     3 │     2 │
└───────┴───────┘

Because of that, it's not equivalent to a positional join when joining across different tables. The rowid should generally only be used within the same table.

rowid columns are also only present in DuckDB tables - we don't emit them for other data sources (yet?).

That being said - the rowid operations will be much faster than positional joins since they are much easier to parallelize.

[Alex-Monahan]

Wow, arbitrary limit/offset queries getting faster would be amazing! This will come in super handy for table sampling.
Tagging @hamilton !

[Maxxen]

@Mytherin added some more tests, but python CI is still messing up

[carlopi]

@Mytherin added some more tests, but python CI is still messing up

This is due to a problem with libxml, that is a postgres_scanner dependency. I am not sure what proper solution could be. If this is frequent (failure mode is network request 503 at about 13-15 minutes) we could remove azure from what's build on regular CI.

[Mytherin]

Thanks! I think this is good to go

[hamilton]

Very excited about this pushdown!

[carlopi]

( I am not sure the stuff I wrote is completely correct about the rewrites being equivalent, there are some caveat that make rowid and LIMIT not the same with deletes / updates, so use with care, but also here very exited )

[FrancoisLepoutre]

Hi all, that looks like a great helpful addition! There are indeed data selection scenarii in which the rowid can help deliver subsets in a smarter way.

IMHO there are others data handling cases in UI mouse-controlled interfaces in which grasping data via some sort of "current id pointer" with a firm and fast handle on the duckdb rowid can make sense as well. In such scenarii, 0.292741 sec. is a nogo. But 0.000455 sec. certainly ain't. No need for an additional buffering layer at app level.