Skip to content

[core] Support projected binary index manifest scans - #9095

Merged
JingsongLi merged 2 commits into
apache:masterfrom
leaves12138:codex/binary-index-manifest-scan
Aug 7, 2026
Merged

[core] Support projected binary index manifest scans#9095
JingsongLi merged 2 commits into
apache:masterfrom
leaves12138:codex/binary-index-manifest-scan

Conversation

@leaves12138

@leaves12138 leaves12138 commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

What changed

  • Add a reusable BinaryIndexManifestEntry view for index manifest records.
  • Add projected binary scan overloads to IndexFileHandler, backed by IndexManifestFile.scan, following the existing BinaryManifestEntry scan pattern.
  • Read only kind, partition, bucket, index type, and the global-index fields needed by planning, while reusing one mutable entry view.
  • Expose schema field-name constants instead of duplicating on-disk field names in projections.
  • Add focused coverage for the public handler API, projected field values, ADD/DELETE records, absent global-index metadata, and iterator reuse/clearing.

Why

Memory-sensitive planning only needs a small subset of each index manifest entry. Existing scans materialize complete IndexManifestEntry and IndexFileMeta object graphs, including fields that these planning paths never consume. A projected binary scan avoids those objects and reuses one mutable entry view across the iterator.

Impact

Existing index manifest serialization and readers are unchanged. The new API is opt-in and keeps the existing on-disk schema intact.

Validation

mvn -pl paimon-core -am -DskipITs -Dfast \
  -Dtest=BinaryIndexManifestEntryTest,IndexManifestEntrySerializerTest,IndexFileHandlerTest \
  -Dsurefire.failIfNoSpecifiedTests=false test

Result: 13 tests passed. Spotless and Checkstyle passed as part of the Maven build.

@leaves12138
leaves12138 marked this pull request as ready for review August 7, 2026 10:01

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit f75c52d into apache:master Aug 7, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants