significant degraded performance of select * from large tables in main compared to datafusion v41.0.0 #12304
Description
Describe the bug
after try latest main, I find the plan performance of select * from large tables is degraded, like below, you can see that physical_select_all_from_1000 has a 5x performance degradation
group main v41
----- ---- ---
logical_aggregate_with_join 1.53 1271.0±21.98µs ? ?/sec 1.00 828.2±25.11µs ? ?/sec
logical_plan_tpcds_all 1.21 156.4±2.05ms ? ?/sec 1.00 129.1±7.42ms ? ?/sec
logical_plan_tpch_all 1.11 14.2±2.75ms ? ?/sec 1.00 12.8±0.31ms ? ?/sec
logical_select_all_from_1000 1.00 4.0±0.24ms ? ?/sec 4.40 17.8±0.19ms ? ?/sec
logical_select_one_from_700 1.01 620.0±13.54µs ? ?/sec 1.00 613.3±36.38µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.18 674.9±14.66µs ? ?/sec 1.00 573.8±13.26µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.11 616.7±19.91µs ? ?/sec 1.00 556.3±19.04µs ? ?/sec
physical_plan_tpcds_all 1.26 1112.4±27.08ms ? ?/sec 1.00 884.2±26.51ms ? ?/sec
physical_plan_tpch_all 1.10 62.8±1.28ms ? ?/sec 1.00 57.3±2.48ms ? ?/sec
physical_plan_tpch_q1 1.12 2.3±0.03ms ? ?/sec 1.00 2.0±0.06ms ? ?/sec
physical_plan_tpch_q10 1.12 3.3±0.17ms ? ?/sec 1.00 2.9±0.35ms ? ?/sec
physical_plan_tpch_q11 1.08 2.6±0.10ms ? ?/sec 1.00 2.4±0.04ms ? ?/sec
physical_plan_tpch_q12 1.08 2.0±0.03ms ? ?/sec 1.00 1860.8±34.67µs ? ?/sec
physical_plan_tpch_q13 1.08 1498.7±39.12µs ? ?/sec 1.00 1389.9±28.17µs ? ?/sec
physical_plan_tpch_q14 1.16 1851.9±91.01µs ? ?/sec 1.00 1594.1±70.39µs ? ?/sec
physical_plan_tpch_q16 1.05 2.5±0.06ms ? ?/sec 1.00 2.4±0.32ms ? ?/sec
physical_plan_tpch_q17 1.06 2.2±0.04ms ? ?/sec 1.00 2.1±0.04ms ? ?/sec
physical_plan_tpch_q18 1.00 2.7±0.04ms ? ?/sec 1.03 2.8±0.47ms ? ?/sec
physical_plan_tpch_q19 1.03 3.9±0.05ms ? ?/sec 1.00 3.8±0.06ms ? ?/sec
physical_plan_tpch_q2 1.14 5.9±0.41ms ? ?/sec 1.00 5.2±0.21ms ? ?/sec
physical_plan_tpch_q20 1.06 3.0±0.05ms ? ?/sec 1.00 2.8±0.05ms ? ?/sec
physical_plan_tpch_q21 1.08 4.3±0.07ms ? ?/sec 1.00 4.0±0.13ms ? ?/sec
physical_plan_tpch_q22 1.07 2.2±0.03ms ? ?/sec 1.00 2.0±0.04ms ? ?/sec
physical_plan_tpch_q3 1.10 2.2±0.12ms ? ?/sec 1.00 2.0±0.15ms ? ?/sec
physical_plan_tpch_q4 1.12 1679.2±47.29µs ? ?/sec 1.00 1501.2±96.77µs ? ?/sec
physical_plan_tpch_q5 1.06 3.1±0.05ms ? ?/sec 1.00 2.9±0.04ms ? ?/sec
physical_plan_tpch_q6 1.05 1000.2±22.01µs ? ?/sec 1.00 956.7±22.22µs ? ?/sec
physical_plan_tpch_q7 1.16 4.3±0.05ms ? ?/sec 1.00 3.7±0.05ms ? ?/sec
physical_plan_tpch_q8 1.19 5.4±0.15ms ? ?/sec 1.00 4.6±0.06ms ? ?/sec
physical_plan_tpch_q9 1.26 4.4±0.09ms ? ?/sec 1.00 3.5±0.05ms ? ?/sec
physical_select_all_from_1000 5.21 220.3±5.03ms ? ?/sec 1.00 42.3±0.38ms ? ?/sec
physical_select_one_from_700 1.00 2.3±0.03ms ? ?/sec 1.00 2.3±0.19ms ? ?/sec
To Reproduce
v41's commit id is b10b820, and main commit id is 4a227c5
git checkout main
cargo bench --bench sql_planner -- --save-baseline main
git checkout v41
cargo bench --bench sql_planner -- --save-baseline v41
critcmp main v41
Expected behavior
No response
Additional context
No response