Repository navigation
[Ideas] Add ORCA intra-segment parallel scan support for partitioned AO tables #2128
Description
For The parallel PostgreSQL plan reduces median execution time from 751.380 to 659.712 ms on unmodified upstream main (12.2% less time; 1.14× speedup) and from 713.039 to 649.303 ms on unmodified upstream REL_2_STABLE (8.9% less time; 1.10× speedup), with The limitation reproduces on both unmodified upstream branches. This appears to be a missing capability in ORCA's intra-segment parallel scan support, rather than a cost-model choice between serial and parallel dynamic scan alternatives. Parallel execution improves this query's elapsed time in these measurements, but does not close the performance gap to GP6 on the unmodified builds; the cross-system comparison is included below. Every timing and plan in this report was collected with Environment and procedure
Cluster startup option and setting verificationSHOW gp_resqueue_priority; -- must report offThe exact restart and sequential measurement procedure is in run_priority_off_comparison.py. Preserve the appropriate dispatch/execute role arguments when restarting nodes. Reproduction SQLCreate and populate this table in each test database where Create, populate and analyze ao13CREATE TABLE ao13 (
id bigint,
d date,
col1 int,
col2 int,
pad text
)
WITH (appendonly=true, orientation=row)
DISTRIBUTED BY (id)
PARTITION BY RANGE (d) (
START ('2020-01-01'::date)
END ('2020-01-14'::date)
EVERY ('1 day'::interval)
);
INSERT INTO ao13
SELECT
g,
'2020-01-01'::date + (g % 13),
g % 97,
g % 89,
repeat('x', 50)
FROM generate_series(1, 6000000) g;
ANALYZE ao13;The SQL for each optimizer mode accompanies its plan below. In Cloudberry, disabling Transaction-scoped fallback reproductionBEGIN;
SET LOCAL optimizer = on;
SET LOCAL optimizer_enable_dynamictablescan = off;
SET LOCAL optimizer_disable_dynamic_table_scan = off;
SET LOCAL enable_parallel = on;
SET LOCAL max_parallel_workers_per_gather = 4;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT count(*) FROM ao13;
COMMIT;Cloudberry unmodified upstream mainSource commit ORCA, parallel onReproduction SQLSET optimizer = on;
SET optimizer_enable_dynamictablescan = on;
SET optimizer_disable_dynamic_table_scan = off;
SET enable_parallel = on;
SET max_parallel_workers_per_gather = 4;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT count(*) FROM ao13;Query execution plan — measured round 5PostgreSQL, parallel offReproduction SQLSET optimizer = off;
SET optimizer_enable_dynamictablescan = on;
SET optimizer_disable_dynamic_table_scan = off;
SET enable_parallel = off;
SET max_parallel_workers_per_gather = 4;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT count(*) FROM ao13;Query execution plan — measured round 5PostgreSQL, parallel onReproduction SQLSET optimizer = off;
SET optimizer_enable_dynamictablescan = on;
SET optimizer_disable_dynamic_table_scan = off;
SET enable_parallel = on;
SET max_parallel_workers_per_gather = 4;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT count(*) FROM ao13;Query execution plan — measured round 5ORCA requested, dynamic scan disabled (fallback)Reproduction SQLSET optimizer = on;
SET optimizer_enable_dynamictablescan = off;
SET optimizer_disable_dynamic_table_scan = off;
SET enable_parallel = on;
SET max_parallel_workers_per_gather = 4;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT count(*) FROM ao13;Query execution plan — measured round 5Cloudberry unmodified upstream REL_2_STABLEOfficial apache/cloudberry REL_2_STABLE commit ORCA, parallel onReproduction SQLSET optimizer = on;
SET optimizer_enable_dynamictablescan = on;
SET optimizer_disable_dynamic_table_scan = off;
SET enable_parallel = on;
SET max_parallel_workers_per_gather = 4;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT count(*) FROM ao13;Query execution plan — measured round 5PostgreSQL, parallel offReproduction SQLSET optimizer = off;
SET optimizer_enable_dynamictablescan = on;
SET optimizer_disable_dynamic_table_scan = off;
SET enable_parallel = off;
SET max_parallel_workers_per_gather = 4;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT count(*) FROM ao13;Query execution plan — measured round 5PostgreSQL, parallel onReproduction SQLSET optimizer = off;
SET optimizer_enable_dynamictablescan = on;
SET optimizer_disable_dynamic_table_scan = off;
SET enable_parallel = on;
SET max_parallel_workers_per_gather = 4;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT count(*) FROM ao13;Query execution plan — measured round 5ORCA requested, dynamic scan disabled (fallback)Reproduction SQLSET optimizer = on;
SET optimizer_enable_dynamictablescan = off;
SET optimizer_disable_dynamic_table_scan = off;
SET enable_parallel = on;
SET max_parallel_workers_per_gather = 4;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT count(*) FROM ao13;Query execution plan — measured round 5Cloudberry 3.0.0-devel + PR #2094PR #2094 was an attempt to address the AO scan performance problem observed in this scenario, so a build including this PR was also benchmarked. It targets tuple-processing overhead through lazy attribute extraction; it does not add parallel Dynamic Seq Scan support. Main-based PR #2094 test build, source commit This is not unmodified upstream main. PR #2094 adds lazy AO MemTuple attribute extraction, which allows ORCA, parallel onReproduction SQLSET optimizer = on;
SET optimizer_enable_dynamictablescan = on;
SET optimizer_disable_dynamic_table_scan = off;
SET enable_parallel = on;
SET max_parallel_workers_per_gather = 4;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT count(*) FROM ao13;Query execution plan — measured round 5PostgreSQL, parallel offReproduction SQLSET optimizer = off;
SET optimizer_enable_dynamictablescan = on;
SET optimizer_disable_dynamic_table_scan = off;
SET enable_parallel = off;
SET max_parallel_workers_per_gather = 4;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT count(*) FROM ao13;Query execution plan — measured round 5PostgreSQL, parallel onReproduction SQLSET optimizer = off;
SET optimizer_enable_dynamictablescan = on;
SET optimizer_disable_dynamic_table_scan = off;
SET enable_parallel = on;
SET max_parallel_workers_per_gather = 4;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT count(*) FROM ao13;Query execution plan — measured round 5ORCA requested, dynamic scan disabled (fallback)Reproduction SQLSET optimizer = on;
SET optimizer_enable_dynamictablescan = off;
SET optimizer_disable_dynamic_table_scan = off;
SET enable_parallel = on;
SET max_parallel_workers_per_gather = 4;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT count(*) FROM ao13;Query execution plan — measured round 5GP6-based OpenGPDBInstalled binary: The table setup and warm-up/five-run protocol match Cloudberry. Cloudberry-specific parallel GUCs are not used. Compiler, assertion and codebase differences prevent interpreting cross-build timings as an isolated GP6-versus-Cloudberry engine regression. ORCAReproduction SQLSET optimizer = on;
SET optimizer_enable_dynamictablescan = on;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT count(*) FROM ao13;Query execution plan — measured round 5PostgreSQLReproduction SQLSET optimizer = off;
SET optimizer_enable_dynamictablescan = on;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT count(*) FROM ao13;Query execution plan — measured round 5Experimental backport of PR #2094 to REL_2_STABLEPR #2094 snapshot The backend and required extensions were rebuilt cleanly with the same compiler and debug/assertion configuration as the clean REL_2_STABLE build, with 32 KiB blocks and 16 MiB shared buffers. Coordinator port 21740; three primary segments on 21741–21743. This experimental backport has a reproduced correctness failure in ORCA UPDATE of an AO table with a stored generated column. It is not a validated release-ready backport. The failure and exact reproduction are documented below; the Direct comparison on REL_2_STABLEThe clean baseline and patched cluster were benchmarked sequentially after compilation and targeted checks finished, each using the same table setup, one warm-up and five measured runs per mode. All eight explicit count checks returned 6000000. This dedicated comparison has its own clean-baseline measurements; use the two columns below together. The laptop/background-load qualification at the start of this issue also applies here.
The ORCA plan remains Experimental backport: query SQL and complete plansORCA, parallel onReproduction SQLSET optimizer = on;
SET optimizer_enable_dynamictablescan = on;
SET optimizer_disable_dynamic_table_scan = off;
SET enable_parallel = on;
SET max_parallel_workers_per_gather = 4;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT count(*) FROM ao13;Query execution plan — measured round 5PostgreSQL, parallel offReproduction SQLSET optimizer = off;
SET optimizer_enable_dynamictablescan = on;
SET optimizer_disable_dynamic_table_scan = off;
SET enable_parallel = off;
SET max_parallel_workers_per_gather = 4;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT count(*) FROM ao13;Query execution plan — measured round 5PostgreSQL, parallel onReproduction SQLSET optimizer = off;
SET optimizer_enable_dynamictablescan = on;
SET optimizer_disable_dynamic_table_scan = off;
SET enable_parallel = on;
SET max_parallel_workers_per_gather = 4;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT count(*) FROM ao13;Query execution plan — measured round 5ORCA requested, dynamic scan disabled (fallback)Reproduction SQLSET optimizer = on;
SET optimizer_enable_dynamictablescan = off;
SET optimizer_disable_dynamic_table_scan = off;
SET enable_parallel = on;
SET max_parallel_workers_per_gather = 4;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT count(*) FROM ao13;Query execution plan — measured round 5Correctness checks and known failureWith ORCA requested, 14 targeted assertions passed before the stored-generated-column UPDATE assertion failed. These checks cover wide rows, NULLs, arrays, large text, partial and late attribute reads, sort/materialization, index lookup, ordinary and distribution-key UPDATE, VACUUM compaction, column-layout changes and table rewrite. Some operations use PostgreSQL fallback, as recorded in the logs. The PostgreSQL-planner run completed all 16 assertions. These targeted checks are not a full regression suite. A separate transaction-scoped reproduction confirmed the difference:
For example, the row Stored generated column UPDATE reproduction\set ON_ERROR_STOP on
SET optimizer = on;
BEGIN;
CREATE TABLE public.backport_generated_check (id int, a int, b int GENERATED ALWAYS AS (a*2) STORED) USING ao_row DISTRIBUTED BY(id);
INSERT INTO public.backport_generated_check (id,a) SELECT g,g FROM generate_series(1,100) g;
EXPLAIN (COSTS OFF) UPDATE public.backport_generated_check SET a=a+1;
UPDATE public.backport_generated_check SET a=a+1;
SELECT count(*) AS rows, count(*) FILTER (WHERE b IS DISTINCT FROM a*2) AS incorrect_generated_values FROM public.backport_generated_check;
SELECT * FROM public.backport_generated_check WHERE b IS DISTINCT FROM a*2 ORDER BY id LIMIT 5;
ROLLBACK;Artifacts: backport patch, compatibility notes, build procedure, comparison runner, clean-baseline plans, clean-baseline measurements, backport plans, backport measurements, targeted SQL, ORCA test log, PostgreSQL test log, generated-column failure log. InterpretationAll tested modes returned the expected count of 6,000,000 rows. ORCA used a serial Dynamic Seq Scan on both upstream branches, while the PostgreSQL planner produced Parallel Append. Expected Cloudberry behaviorWhen parallel execution is enabled and safe, ORCA should be able to consider a parallel alternative for scanning partitioned AO tables and choose it when justified by cost. The required outcome is parallel work distribution without changing query results or partition-pruning semantics. This could be implemented as a parallel-aware DynamicSeqScan or an appropriate Parallel Append plan; a particular plan-node representation is not required by this report. Cloudberry code investigationThe following observations come from inspection of the local Cloudberry checkout:
Potential implementation directionsFor a parallel-aware DynamicSeqScan, a shared partition queue/counter could distribute whole partitions among workers within each segment. This would require coordinated partition-pruning results, DSM initialization and worker attachment, correct rescan behavior, and corresponding ORCA/DXL/plan-translation support. The ORCA implementation must also account for worker distribution, rewindability, and cost. Whole-partition scheduling is a limited first step: it cannot parallelize a scan when pruning leaves only one partition. Sharing work within a partition would require additional support. AO scan granularity also matters. In the inspected An alternative is to extend ORCA's existing dynamic-scan-to-Append translation to generate a valid parallel plan and reuse the Parallel Append executor infrastructure. Related discussions and PRs
Suggested validation for a fix
Final comparison — gp_resqueue_priority=off on every nodeAll rows use one warm-up and five measured runs; median excludes warm-up.
The experimental backport has a dedicated paired baseline and known correctness failure documented above. Execution time relative to GP6The reference medians are 442.338 ms for GP6 ORCA and 461.166 ms for the GP6 PostgreSQL planner. The table shows Cloudberry execution time / GP6 execution time: a ratio above 1 means Cloudberry took longer; a ratio below 1 means it took less time. ORCA is compared with GP6 ORCA. All PostgreSQL execution modes, including ORCA fallback, are compared with the GP6 PostgreSQL planner. GP6 uses serial Append, so the parallel columns compare the available execution strategies rather than identical plan shapes.
In these measurements, unmodified main and REL_2_STABLE took 1.70× and 1.61× as long as GP6 with ORCA, respectively. Even with PostgreSQL Parallel Append, they took 1.43× and 1.41× as long as the GP6 PostgreSQL planner. The PR #2094 builds narrow the ORCA gap and show lower median times than GP6 in the PostgreSQL parallel mode. These are provisional laptop measurements with the build differences documented above, not an isolated cross-version regression measurement. The experimental REL_2_STABLE backport also has a known UPDATE correctness failure. Patch that I used for measures REL_2_STABLE + #2094 attached as a file. Related issuesAdd ORCA intra-segment parallel scan support for partitioned AO tables Are you willing to submit a PR?
|
Replies: 1 comment 1 reply
|
Thanks for the very thorough report, @Alena0704. Your code reading matches what is on main today, and the conclusion is right: this is a missing capability in ORCA, not a cost-model decision. Current state
I already have a draft PR for this and will open it shortly. It follows the second direction you described: reuse the dynamic-scan-to-Append translation and the existing Parallel Append executor, instead of making DynamicSeqScan parallel-aware. The main reasons:
|
Thanks for the very thorough report, @Alena0704. Your code reading matches what is on main today, and the conclusion is right: this is a missing capability in ORCA, not a cost-model decision.
Current state