Skip to content

[flink] Avoid slot deadlock when sorting normalized index keys - #9394

Merged
JingsongLi merged 1 commit into
apache:masterfrom
leaves12138:flink/multivalue-batch-exchange
Aug 26, 2026
Merged

[flink] Avoid slot deadlock when sorting normalized index keys#9394
JingsongLi merged 1 commit into
apache:masterfrom
leaves12138:flink/multivalue-batch-exchange

Conversation

@leaves12138

Copy link
Copy Markdown
Contributor

Purpose

Non-identity sorted indexes, such as multivalue indexes, normalize one source value into multiple index keys. The normalized-key path used keyBy before the heap-and-spill sorter. Replacing it with a custom partitioner avoids Flink inserting a redundant managed-memory sorter, but a pipelined exchange can require the reader and sorter stages to acquire slots concurrently. A constrained batch deployment may then make no progress once the upstream network buffers fill.

Changes

  • Partition normalized index keys by build task without keyBy.
  • Explicitly use StreamExchangeMode.BATCH, matching the deadlock-breaking approach used by range shuffle.
  • Preserve the existing build-task co-location semantics.
  • Add tests for the partition mapping and exchange mode.

BTree and scalar bitmap indexes use the identity/global-sort path and are unaffected.

Tests

  • SortedIndexTopoBuilderTest
  • SortedGlobalIndexITCase#testMultiValueIndex
  • A constrained-slot validation with 1,000,006 rows completed with full multivalue-index coverage and content checks.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit 3130869 into apache:master Aug 26, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants