Skip to content

Pipe/Load: Implement multi-disk awareness of multiple file systems during file copying and moving#15356

Merged
Caideyipi merged 18 commits intoapache:masterfrom
luoluoyuyu:multi-disk
Jul 28, 2025
Merged

Pipe/Load: Implement multi-disk awareness of multiple file systems during file copying and moving#15356
Caideyipi merged 18 commits intoapache:masterfrom
luoluoyuyu:multi-disk

Conversation

@luoluoyuyu
Copy link
Member

@luoluoyuyu luoluoyuyu commented Apr 17, 2025

Description

As the title said


This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods.
  • added or updated version, license, or notice information
  • added comments explaining the "why" and the intent of the code wherever would not be obvious
    for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold
    for code coverage.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR

@luoluoyuyu luoluoyuyu marked this pull request as draft April 17, 2025 08:44
@SteveYurongSu SteveYurongSu self-assigned this Apr 17, 2025
@luoluoyuyu luoluoyuyu marked this pull request as ready for review April 22, 2025 12:23
# Conflicts:
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/load/LoadTsFileAnalyzer.java
@SteveYurongSu
Copy link
Member

@Pengzna PTAL

Copy link
Collaborator

@Pengzna Pengzna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job! Only a few reviews, PTAL

luoluoyuyu added 5 commits May 9, 2025 16:28
# Conflicts:
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/load/LoadTsFileAnalyzer.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/disk/ILoadDiskSelector.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/disk/InheritSystemMultiDisksStrategySelector.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/disk/MinIOSelector.java
@Caideyipi Caideyipi merged commit 432e027 into apache:master Jul 28, 2025
26 of 27 checks passed
luoluoyuyu added a commit to luoluoyuyu/iotdb that referenced this pull request Jul 29, 2025
…ring file copying and moving (apache#15356)

* Pipe/Load: Implement multi-disk awareness of multiple file systems during file copying and moving

* spotless

* fix

* fix

* update ActiveLoadUtil

* update InheritSystemMultiDisksStrategySelector

* update ILoadDisksSelector

* update ActiveLoadUtil

* fix

* fix

* modify the code based on review comments

* modify the code based on review comments

* spotless

* fix

* fix

(cherry picked from commit 432e027)

# Conflicts:
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/receiver/protocol/thrift/IoTDBDataNodeReceiver.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/load/LoadTsFileAnalyzer.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/disk/ILoadDiskSelector.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/disk/InheritSystemMultiDisksStrategySelector.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/disk/MinIOSelector.java
@luoluoyuyu luoluoyuyu deleted the multi-disk branch July 31, 2025 01:55
Caideyipi pushed a commit that referenced this pull request Aug 4, 2025
…le systems during file copying and moving (#15356) (#16052)

* Pipe/Load: Implement multi-disk awareness of multiple file systems during file copying and moving (#15356)

* Pipe/Load: Implement multi-disk awareness of multiple file systems during file copying and moving

* spotless

* fix

* fix

* update ActiveLoadUtil

* update InheritSystemMultiDisksStrategySelector

* update ILoadDisksSelector

* update ActiveLoadUtil

* fix

* fix

* modify the code based on review comments

* modify the code based on review comments

* spotless

* fix

* fix

(cherry picked from commit 432e027)

# Conflicts:
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/receiver/protocol/thrift/IoTDBDataNodeReceiver.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/load/LoadTsFileAnalyzer.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/disk/ILoadDiskSelector.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/disk/InheritSystemMultiDisksStrategySelector.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/disk/MinIOSelector.java

* fix
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants