Skip to content

[core] Split chain table batch scans by key range for read parallelism. - #8831

Merged
JingsongLi merged 1 commit into
apache:masterfrom
Stephen0421:chain-table-key-range-split
Jul 26, 2026
Merged

[core] Split chain table batch scans by key range for read parallelism.#8831
JingsongLi merged 1 commit into
apache:masterfrom
Stephen0421:chain-table-key-range-split

Conversation

@Stephen0421

Copy link
Copy Markdown
Contributor

Purpose

Chain table batch reads currently produce one split per bucket, which limits read parallelism for large buckets. This PR adds key-range split generation for chain table batch scans:

  • Introduces chain-table.split.key-range-enabled (default: true); set to false to fall back to one split per bucket
  • Batch scans use IntervalPartition + BinPacking to split snapshot and delta files by key range; files with overlapping key ranges always stay in the same split so all versions of a key across branches are merged correctly
  • Streaming reads are unchanged and still use the original one-split-per-bucket behavior

Tests

  • Added ChainTableUtilsTest.testBuildChainSplitsWithKeyRangeSplitting: verifies key-disjoint sections are split into separate splits and overlapping snapshot/delta files are kept together
  • Added ChainTableUtilsTest.testBuildChainSplitsKeyRangeSplittingLargeTargetKeepsOneSplit: verifies a large targetSplitSize packs all sections into a single split
  • Added ChainTableUtilsTest.testBuildChainSplitsWithoutKeyRangeSplitting: verifies a null keyComparator preserves the original one-split-per-bucket behavior

@Stephen0421
Stephen0421 force-pushed the chain-table-key-range-split branch from 41bf23d to aad7514 Compare July 23, 2026 11:17
@JingsongLi

Copy link
Copy Markdown
Contributor

After adding chain-table.split.key-range-enabled, docs/generated/core_configuration.html was not updated.

Add chain-table.split.key-range-enabled to split snapshot and delta files
into multiple splits per bucket while keeping overlapping key ranges together.
@Stephen0421
Stephen0421 force-pushed the chain-table-key-range-split branch from aad7514 to e29dc37 Compare July 23, 2026 13:08
@Stephen0421

Copy link
Copy Markdown
Contributor Author

After adding chain-table.split.key-range-enabled, docs/generated/core_configuration.html was not updated.

OK, fix it.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit 4badbf6 into apache:master Jul 26, 2026
13 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants