[Ideas] Better parallel scans for AO and AOCS tables

[roseduan]

Description

Hi all,

I'm working on making parallel scan of append-optimized tables (AO and AOCS) more useful, and I'd like your feedback.

The problem
Today each parallel worker reads a whole segment file. A table with few segment files, often just one after a bulk load, gets little or no parallelism(if GUC gp_appendonly_insert_files is not set), and one big file is always read by a single worker.

The idea
Split each segment file into many small chunks of rows, and let workers take chunks one at a time from a shared counter. Every worker stays busy until the scan ends, however many segment files the table has. No index or extra metadata is needed.

Status

• AOCS: implemented. Results match serial scans, the related regression tests pass, and large tables scan several times faster in my tests. The planner can now choose a parallel plan for AOCS tables with only one segment file.
• AO row: planned next, using the same approach.

Questions for the community

• Does this direction make sense to you?
• Anything else I should cover or test?

Thanks!

Use case/motivation

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[Alena0704]

This looks promising, especially for tables with a single large segment file. Could you explain how a worker reaches its assigned row chunk without an index or additional metadata? Does it retain its position and skip block headers, and can multiple workers end up reading or decompressing the same blocks?
Could you also share the patch and benchmark setup—table size, compression, projected columns, worker count, and whether the data was cached? That would help assess the tradeoff between better balancing and extra scan work.

[roseduan]

This looks promising, especially for tables with a single large segment file. Could you explain how a worker reaches its assigned row chunk without an index or additional metadata? Does it retain its position and skip block headers, and can multiple workers end up reading or decompressing the same blocks? Could you also share the patch and benchmark setup—table size, compression, projected columns, worker count, and whether the data was cached? That would help assess the tradeoff between better balancing and extra scan work.

Thanks for your attention. Well, may be the saying 'without additional metadata' is not very accurate. The whole mechanism is roughly like:

• get how many tuples of an aocs table
• calculate a number of rows that a single worker will scan (this is called a chunk, a chunk contains multiple rows)
• each worker scan a chunk at a time, and then claim another chunk to scan
• since we don't have a block directory or any other information to find the start row quickly, so right now we only scan the block header to find out if the start row is in the current block. (but we only scan the block header from last position instead of the begining, so maybe it's not an expensive cost, but I will test it anyway)

In addition, I will do some benchmarks later, so you can run the new implementation in your env. Thanks again, feel free to give me any feedback.

[yjhjstz]

Can you provide tpch bench compared to old parallel aocs ?

[roseduan]

Can you provide tpch bench compared to old parallel aocs ?

Ok, I will do a full TPCH and TPCDS benchmark later, and I will share the results. Thanks.