Late Materialization Optimizer

[Mytherin]

This PR adds the Late Materialization optimizer that enables late materialization for certain queries - in particular top-n (ORDER BY .. LIMIT ..), limit + offset, and sample queries. The optimization piggy-backs off of the row-id filter pushdown introduced in #15020 - and does the row-id rewrites mentioned there automatically.

Rewrites

Here are some examples of rewrites:

Top-N

SELECT * FROM lineitem ORDER BY l_extendedprice DESC LIMIT 5;

-- becomes

SELECT * FROM lineitem WHERE rowid IN (
    SELECT rowid FROM lineitem ORDER BY l_extendedprice DESC LIMIT 5)
ORDER BY l_extendedprice DESC;

Limit + Offset

SELECT * FROM lineitem LIMIT 1000000 OFFSET 10000000;

-- becomes


SELECT * FROM lineitem WHERE rowid IN (
    SELECT rowid FROM lineitem LIMIT 1000000 OFFSET 10000000)
ORDER BY rowid;

Performance

SELECT * FROM lineitem ORDER BY l_extendedprice DESC LIMIT 5;

v1.1.3	main	new
2.1s	0.07s	0.01s

SELECT * FROM lineitem ORDER BY l_orderkey DESC LIMIT 5;

v1.1.3	main	new
10.2s	2.8s	0.14s

SELECT * FROM lineitem LIMIT 1000000 OFFSET 10000000;

v1.1.3	main	new
0.46s	0.46s	0.07s

[abramk]

@Mytherin I tried your PR against the following and am not seeing a speed up.

 create table tbl1(pk LONG PRIMARY KEY,
        col_long_text VARCHAR,
        col_long_text_2 VARCHAR,
        col_truncated VARCHAR,
        col_sort LONG);

 INSERT INTO tbl1 SELECT
        i AS pk,
        rpad('', 64000, 'A') AS col_long_text,
        rpad('', 64000, 'B') AS col_long_text_2,
        rpad('', 4000, 'A') AS col_truncated,
        floor(random() * 9223372036854775807)::long AS col_sort FROM generate_series(1, 50000) s(i);

I then run this query:

 select * from tbl1 order by pk desc limit 1000;

and it takes 1.477 s but when I run the transformed version manually:

select * from tbl1 where rowid in (select rowid from tbl1 order by pk desc limit 1000);

I see it takes 155ms.

I think I have your PR compiled correctly as PRAGMA profiling_mode = 'detailed' shows:

┌────────────────────────────────────────────────┐
│               Optimizer: 0.0002s               │
│┌──────────────────────────────────────────────┐│
││        Build Side Probe Side: 0.0000s        ││
││           Column Lifetime: 0.0000s           ││
││           Common Aggregate: 0.0000s          ││
││        Common Subexpressions: 0.0000s        ││
││      Compressed Materialization: 0.0000s     ││
││          Cte Filter Pusher: 0.0000s          ││
││             Deliminator: 0.0000s             ││
││           Duplicate Groups: 0.0000s          ││
││         Empty Result Pullup: 0.0000s         ││
││         Expression Rewriter: 0.0000s         ││
││              Extension: 0.0000s              ││
││            Filter Pullup: 0.0000s            ││
││           Filter Pushdown: 0.0000s           ││
││              In Clause: 0.0000s              ││
││         Join Filter Pushdown: 0.0000s        ││
││              Join Order: 0.0000s             ││
││         Late Materialization: 0.0000s        ││
││            Limit Pushdown: 0.0000s           ││
││           Materialized Cte: 0.0000s          ││
││             Regex Range: 0.0000s             ││
││            Reorder Filter: 0.0000s           ││
││          Sampling Pushdown: 0.0000s          ││
││        Statistics Propagation: 0.0000s       ││
││             Sum Rewriter: 0.0000s            ││
││                Top N: 0.0000s                ││
││           Unnest Rewriter: 0.0000s           ││
││            Unused Columns: 0.0000s           ││
│└──────────────────────────────────────────────┘│
└────────────────────────────────────────────────┘

You can see the Late Materialization filter in the list.

[Mytherin]

That's because of the limit size. The late materialization optimizer only triggers when the limit is <= 50 at this point (see max_row_count in the code). I can make this configurable in a follow-up PR.

[carlopi]

Minor, also to be done on as subsequent PR, there are 3 warnings turned errors in the amalgamation run (OSX Debug):

-----------------------
-- Writing src/amalgamation/duckdb.hpp --
-----------------------
------------------------
-- Writing src/amalgamation/duckdb.cpp --
------------------------
duckdb.cpp:253075:11: error: redundant move in return statement [-Werror,-Wredundant-move]
                        return std::move(op);
                               ^
duckdb.cpp:253075:11: note: remove std::move call here
                        return std::move(op);
                               ^~~~~~~~~~  ~
duckdb.cpp:2[53](https://github.com/Mytherin/duckdb/actions/runs/12753535210/job/35545445957#step:10:54)086:11: error: redundant move in return statement [-Werror,-Wredundant-move]
                        return std::move(op);
                               ^
duckdb.cpp:253086:11: note: remove std::move call here
                        return std::move(op);
                               ^~~~~~~~~~  ~
duckdb.cpp:253099:11: error: redundant move in return statement [-Werror,-Wredundant-move]
                        return std::move(op);
                               ^
duckdb.cpp:253099:11: note: remove std::move call here
                        return std::move(op);
                               ^~~~~~~~~~  ~
3 errors generated.

[YuweiXiao]

hey! wondering if this optimization applies for non-duckdb table, e.g., a delta lake table. Thanks!