Skip to content

Pipe: aggregate tablets with different measurements under the same device before write into tsfile#15193

Merged
SteveYurongSu merged 7 commits intoapache:masterfrom
VGalaxies:tsfile-builder
Apr 17, 2025
Merged

Pipe: aggregate tablets with different measurements under the same device before write into tsfile#15193
SteveYurongSu merged 7 commits intoapache:masterfrom
VGalaxies:tsfile-builder

Conversation

@VGalaxies
Copy link
Contributor

As title.

@SteveYurongSu SteveYurongSu self-assigned this Mar 26, 2025
@VGalaxies VGalaxies marked this pull request as ready for review March 29, 2025 09:20
@SteveYurongSu
Copy link
Member

@luoluoyuyu PTAL

@SteveYurongSu SteveYurongSu requested a review from Copilot March 31, 2025 08:53
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces logic to aggregate tablets with matching measurement criteria before writing them into tsfiles, which is aimed at optimizing the writing process for devices with various measurement data.

  • Added a new method, tryBestToAggregateTablets, to aggregate tablets while maintaining consistent timestamps, row sizes, and maximum row numbers.
  • Updated the logic in tryBestToWriteTabletsIntoOneFile to leverage tablet aggregation.
  • Streamlined lambda expressions in integration tests for both push and pull consumer scenarios.

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/util/builder/PipeTreeModelTsFileBuilder.java Added aggregation logic and incorporated it into the tablet writing process.
integration-test/src/test/java/org/apache/iotdb/subscription/it/triple/regression/pushconsumer/multi/IoTDBConsumer2With1TopicShareProcessTsfileIT.java Simplified lambda assertion blocks.
integration-test/src/test/java/org/apache/iotdb/subscription/it/triple/regression/pullconsumer/multi/IoTDBConsumer2With1TopicShareProcessTsfileIT.java Simplified lambda assertion blocks.
Comments suppressed due to low confidence (1)

iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/util/builder/PipeTreeModelTsFileBuilder.java:199

  • [nitpick] Consider renaming the method tryBestToAggregateTablets to a more descriptive name such as aggregateTabletsMatchingCriteria to better convey its purpose.
private Tablet tryBestToAggregateTablets(

@SteveYurongSu SteveYurongSu merged commit e545bf0 into apache:master Apr 17, 2025
50 checks passed
VGalaxies added a commit to VGalaxies/iotdb that referenced this pull request Apr 17, 2025
@VGalaxies VGalaxies deleted the tsfile-builder branch April 17, 2025 03:42
SteveYurongSu pushed a commit that referenced this pull request Apr 21, 2025
JackieTien97 pushed a commit that referenced this pull request May 14, 2025
…vice before write into tsfile (#15193)

(cherry picked from commit e545bf0)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants