Skip to content

[core][flink][spark] Support default bucket number for postpone writes - #9067

Merged
JingsongLi merged 1 commit into
apache:masterfrom
Zouxxyy:xinyu/postpone-default-bucket-direct-write
Aug 6, 2026
Merged

[core][flink][spark] Support default bucket number for postpone writes#9067
JingsongLi merged 1 commit into
apache:masterfrom
Zouxxyy:xinyu/postpone-default-bucket-direct-write

Conversation

@Zouxxyy

@Zouxxyy Zouxxyy commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Purpose

  • Make postpone.default-bucket-num optional. When configured, use its value exactly for partitions receiving their first real-bucket layout and for fixed-bucket overwrite writes.
  • Let Spark write directly to fixed buckets for overwrite operations and snapshots without real buckets, skipping temporary uncommitted bucket -2 files. Appends to existing real-bucket layouts remain on the staged rescale path; mixed batches stay staged and use the configured default only for new partitions.
  • Centralize snapshot-bound bucket-number resolution in Core and reuse it in Spark and Flink. When no default is configured, estimate from postpone.target-row-num-per-bucket or postpone.target-size-per-bucket; execution parallelism no longer determines the logical bucket count.
  • Avoid unnecessary merge-on-read metadata work: plans without selected postpone splits create an empty router without scanning bucket metadata, while other plans load bucket counts and postpone statistics only for partitions containing selected postpone splits.

Tests

  • Core PostponeUtilsTest and TableScanTest: 35 passed
  • Spark PostponeBucketTableTest: 30 passed
  • Flink PostponeBucketTableITCase#testCompactWithTargetRowNumPerBucket: 1 passed
  • mvn spotless:check -Pspark3

@Zouxxyy
Zouxxyy marked this pull request as draft August 6, 2026 07:38
Use an explicitly configured bucket number for safe direct Spark writes to new layouts and overwrite operations. Centralize postpone bucket resolution across Core, Flink, and Spark, decouple logical buckets from engine parallelism, and limit merge-on-read metadata scans to relevant partitions.

Co-Authored-By: Claude Code <noreply@anthropic.com>
AI-Model: gpt-5
AI-Contributed/Feature: 531/531
AI-Contributed/UT: 296/296
@Zouxxyy
Zouxxyy force-pushed the xinyu/postpone-default-bucket-direct-write branch from e361a67 to 74bb6e7 Compare August 6, 2026 09:43
@Zouxxyy Zouxxyy changed the title [core][flink][spark] Use default bucket number for postpone writes [core][flink][spark] Support default bucket number for postpone writes Aug 6, 2026
@Zouxxyy
Zouxxyy marked this pull request as ready for review August 6, 2026 09:49
@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit 50bfb69 into apache:master Aug 6, 2026
14 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants