[IOTDB-2807]Speed up the cross space compaction by multi-threads by choubenson · Pull Request #5415 · apache/iotdb

choubenson · 2022-04-05T07:38:11Z

The cross space comapction execution performance of 0.13.0 is slower than 0.12.5.

Need to speed up.

We achieve parallel reading and writing at the measurement level by adding threads.

qiaojialin · 2022-04-05T08:12:27Z

server/src/main/java/org/apache/iotdb/db/engine/compaction/CompactionUtils.java

+      try {
+        futures.get(i).get();
+      } catch (InterruptedException | ExecutionException e) {
+        logger.error("SubCompactionTask meet errors ", e);


Thread.interrepted().

server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/AbstractCompactionWriter.java

...r/src/main/java/org/apache/iotdb/db/engine/compaction/writer/CrossSpaceCompactionWriter.java

...r/src/main/java/org/apache/iotdb/db/engine/compaction/writer/InnerSpaceCompactionWriter.java

server/src/main/java/org/apache/iotdb/db/engine/compaction/CompactionUtils.java

…HashMap

THUMarkLau

Do we need to have a performance report for this PR before it is merged?

THUMarkLau · 2022-04-05T11:27:53Z

server/src/main/java/org/apache/iotdb/db/engine/compaction/CompactionUtils.java

-      } catch (PathNotExistException e) {
-        logger.info("A deleted path is skipped: {}", e.getMessage());
-        continue;
+      if (measurementsForEachSubTask[idx % subTaskNums] == null) {


Is it necessary to hash here? If the hash is not uniform, then the effect of multithreading could not be worse. Why not just use a concurrency stack or queue, each sub-thread get a measurement to compact when it is available.

Each sub task has its own hashset, so this won't happen.

THUMarkLau · 2022-04-05T11:28:43Z

server/src/main/java/org/apache/iotdb/db/engine/compaction/CompactionUtils.java

+      } catch (InterruptedException | ExecutionException e) {
+        logger.error("SubCompactionTask meet errors ", e);
+        Thread.interrupted();
+        throw new IOException(e);


Throw a InterruptedException instead.

…ent thread

…r, and the following threads will directly flush Chunk

qiaojialin · 2022-04-08T05:15:39Z

...r/src/main/java/org/apache/iotdb/db/engine/compaction/writer/CrossSpaceCompactionWriter.java

+    int fileIndex = seqFileIndexArray[subTaskId];
+    writeRateLimit(chunkWriterMap.get(subTaskId).estimateMaxSeriesMemSize());
+    while (hasTargetFileStartChunkGroup[fileIndex].get() == 1) {
+      // wait until the target file has finished flushing chunk group header


wait and notifyall

qiaojialin · 2022-04-09T06:00:13Z

tsfile/src/main/java/org/apache/iotdb/tsfile/write/writer/TsFileIOWriter.java

    out.flush();
  }

+  public void truncate(long truncateSize) throws IOException {


Suggested change

public void truncate(long truncateSize) throws IOException {

public void truncate(long offset) throws IOException {

…chunk group

…er_meta * remotes/upstream/master: [IOTDB-2872] Rename vsg to dataRegion (apache#5470) [IOTDB-2806][InfluxDB] Compatibility of Apache IoTDB with InfluxDB - Complete UserGuide (apache#5351) [IOTDB-2658] Generate logical plan for query statement —— UT & Raw Data Query & Aggregation Query (apache#5469) [IOTDB-1614] New WAL (apache#5320) Update ContributeGuide.md (apache#5463) user SerializeDeserializeUtil in confignode/AuthorPlan (apache#5467) Modify the jdbc query time column to be empty (apache#5281) [IOTDB-2827] Batch insert in new cluster (apache#5412) [IOTDB-2801] New storage engine framework (apache#5357) [IOTDB-2807]Speed up the cross space compaction by multi-threads (apache#5415) Fix coverage check [IOTDB-2864] Fix Read-only occurred when insert Text values to aligned timeseries (apache#5460) [IOTDB-2803]add AlterTimeSeriesNode and CreateAlignedTimeSeriesNode to PlanNodeType and its serialize and deserialize (apache#5444) [IOTDB-2841] add permission manager model (apache#5401) # Conflicts: # server/src/main/java/org/apache/iotdb/db/mpp/sql/analyze/Analyzer.java # server/src/main/java/org/apache/iotdb/db/mpp/sql/planner/DistributionPlanner.java # server/src/main/java/org/apache/iotdb/db/mpp/sql/planner/LogicalPlanner.java # server/src/main/java/org/apache/iotdb/db/mpp/sql/planner/plan/PlanFragment.java # server/src/main/java/org/apache/iotdb/db/mpp/sql/planner/plan/node/PlanNodeType.java # server/src/main/java/org/apache/iotdb/db/mpp/sql/planner/plan/node/metedata/read/DevicesMetaScanNode.java # server/src/main/java/org/apache/iotdb/db/mpp/sql/planner/plan/node/source/SeriesScanNode.java # server/src/main/java/org/apache/iotdb/db/mpp/sql/planner/plan/node/source/SourceNode.java # server/src/main/java/org/apache/iotdb/db/mpp/sql/statement/StatementVisitor.java # server/src/test/java/org/apache/iotdb/db/mpp/sql/plan/LogicalPlannerTest.java

choubenson added 4 commits April 1, 2022 17:27

first complete concurrent

3612543

update device start time and end time after compacting

ebbd04f

Merge branch 'master' into speedUpCompaction

91b8c1c

fix conflict

8787bb7

qiaojialin reviewed Apr 5, 2022

View reviewed changes

choubenson added 2 commits April 5, 2022 19:05

fix chunkReaderTest and use AtomicInteger array instead of Concurrent…

6254f86

…HashMap

add Thread.interrupted

f304031

THUMarkLau requested changes Apr 5, 2022

View reviewed changes

choubenson added 8 commits April 6, 2022 18:42

fix hasStartChunk param in CrossSpaceCompactionWriter safe in concurr…

4485587

…ent thread

change AtomicInteger[] to ConcurrentHashMap

3c26be1

fix concurrency bug: The first thread has not flushed ChunkGroupHeade…

386479a

…r, and the following threads will directly flush Chunk

change concurrenctHashMap to int[] array

144775c

Merge branch 'master' into speedUpCompaction

86a3c30

merge master

0053bf0

rename map to array

3295ac7

remove status in TsFileResource

fe78277

qiaojialin reviewed Apr 8, 2022

View reviewed changes

choubenson added 4 commits April 8, 2022 15:21

start chunkGroup in all target files and clean empty chunkGroup at last

9d77ab7

spotless

3eeafb9

add flushChunkToWriter method

d16416f

define ChunkWriters as array instead of concurrenctHashMap

c7966bf

qiaojialin reviewed Apr 9, 2022

View reviewed changes

add tests to test whether cross compaction target file contain empty …

65bccce

…chunk group

qiaojialin approved these changes Apr 9, 2022

View reviewed changes

qiaojialin merged commit 99c824f into apache:master Apr 9, 2022

choubenson mentioned this pull request Apr 11, 2022

Cherry pick compaction from master to rel/0.13 #5480

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[IOTDB-2807]Speed up the cross space compaction by multi-threads#5415

[IOTDB-2807]Speed up the cross space compaction by multi-threads#5415
qiaojialin merged 19 commits intoapache:masterfrom
choubenson:speedUpCompaction

choubenson commented Apr 5, 2022

Uh oh!

qiaojialin Apr 5, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

THUMarkLau left a comment

Uh oh!

THUMarkLau Apr 5, 2022

Uh oh!

choubenson Apr 7, 2022

Uh oh!

THUMarkLau Apr 5, 2022

Uh oh!

choubenson Apr 6, 2022

Uh oh!

qiaojialin Apr 8, 2022

Uh oh!

qiaojialin Apr 9, 2022

Uh oh!

choubenson Apr 9, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	public void truncate(long truncateSize) throws IOException {
	public void truncate(long offset) throws IOException {

Conversation

choubenson commented Apr 5, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

THUMarkLau left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants