[IOTDB-1614] New WAL by HeimingZ · Pull Request #5320 · apache/iotdb

HeimingZ · 2022-03-22T14:19:43Z

# Conflicts: # node-commons/src/main/java/org/apache/iotdb/commons/conf/IoTDBConstant.java # server/src/main/java/org/apache/iotdb/db/engine/StorageEngine.java # server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupInfo.java # server/src/main/java/org/apache/iotdb/db/engine/storagegroup/VirtualStorageGroupProcessor.java # server/src/main/java/org/apache/iotdb/db/qp/physical/PhysicalPlan.java # server/src/main/java/org/apache/iotdb/db/qp/physical/crud/InsertRowPlan.java # server/src/main/java/org/apache/iotdb/db/service/metrics/MetricsService.java # server/src/main/java/org/apache/iotdb/db/wal/recover/file/TsFilePlanRedoer.java # server/src/main/java/org/apache/iotdb/db/writelog/manager/MultiFileLogNodeManager.java # server/src/main/java/org/apache/iotdb/db/writelog/node/ExclusiveWriteLogNode.java # server/src/test/java/org/apache/iotdb/db/writelog/IoTDBLogFileSizeTest.java # server/src/test/java/org/apache/iotdb/db/writelog/PerformanceTest.java # server/src/test/java/org/apache/iotdb/db/writelog/recover/DeviceStringTest.java # server/src/test/java/org/apache/iotdb/db/writelog/recover/LogReplayerTest.java # server/src/test/java/org/apache/iotdb/db/writelog/recover/RecoverResourceFromReaderTest.java # server/src/test/java/org/apache/iotdb/db/writelog/recover/SeqTsFileRecoverTest.java # server/src/test/java/org/apache/iotdb/db/writelog/recover/UnseqTsFileRecoverTest.java

wangchao316

do not have permission to open the document. advised to put this doc to jira or confluence.

HeimingZ · 2022-03-23T08:19:31Z

do not have permission to open the document. advised to put this doc to jira or confluence.

I have updated the read authority. It can be seen now. @wangchao316

wangchao316 · 2022-03-24T08:51:51Z

server/src/assembly/resources/conf/iotdb-engine.properties

+# The default value 0 means the storage space will not be controlled.
+# Notice: If this parameter is too small, the write performance may decline.
+# Datatype: int
+# wal_storage_space_in_mb=0


this default parameter is 0 ?

Default 0 means the storage space will controlled by iotdb itself, you can see delete method of WALManager. Besides, related functions of this parameter is still under development.

wangchao316 · 2022-03-24T08:52:45Z

server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java

+  private String timestampPrecision = "ms";

-  private boolean enableDiscardOutOfOrderData = false;
+  // region Write Ahead Log Configuration


this context?

Just remove it from the region of wal parameters, you can still see it in IoTDBConfig L777-L778

wangchao316 · 2022-03-24T09:02:27Z

server/src/main/java/org/apache/iotdb/db/engine/memtable/AbstractMemTable.java


+  @Override
+  public int getMemTableId() {
+    return memTableId;


memTableId is a fixed value, memTableIdCounter.getAndIncrement() == 0.

Each memtable has its own id, this value is initialized at AbstractMemTable L87-L88

wangchao316 · 2022-03-24T09:24:35Z

server/src/main/java/org/apache/iotdb/db/wal/WALManager.java

+  /** help allocate node for users */
+  private int nodeCursor = -1;
+  /** each wal node has a unique long value identifier */
+  private long nodeIdCounter = -1;


nodeIdCounter should use AtomicLong.

nodeCursor and nodeIdCounter are both protected by the nodesLock of WALManager.

wangchao316 · 2022-03-24T09:28:47Z

server/src/main/java/org/apache/iotdb/db/wal/buffer/WALBuffer.java

+
+  private void allocateBuffers() {
+    try {
+      workingBuffer = ByteBuffer.allocateDirect(WAL_BUFFER_SIZE / 2);


Can the number of ByteBuffer.allocateDirect be controlled? If the number of ByteBuffer.allocateDirect is infinite, the memory will be used up?

The memory usage is controlled by the max number of wal node (each node has a wal buffer), whose corresponding parameter is max_wal_num. You can see wal nodes are reused in WALManger.

wangchao316 · 2022-03-24T09:31:54Z

server/src/main/java/org/apache/iotdb/db/wal/buffer/AbstractWALBuffer.java

+      logger.info("create folder {} for wal buffer-{}.", logDirectory, identifier);
+    }
+    currentLogWriter =
+        new WALWriter(


super class of WALWriter is AbstractWALBuffer, why new WALWriter in AbstractWALBuffer ?

Because wal segemented buffer, another type of buffer, can reuse these code. And WALSegementedBuffer is still under development.

Sorry, i misunderstand you. The super class of WALWriter is LogWriter, not AbstractWALBuffer.

wangchao316 · 2022-03-24T09:33:15Z

server/src/main/java/org/apache/iotdb/db/wal/buffer/WALBuffer.java

+      buffersLock.unlock();
+    }
+  }
+  // endregion


Get lock at the beginning of this method, so it's unsafe to delete lock release code.

server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileProcessor.java

HTHou · 2022-03-28T09:50:21Z

server/src/main/java/org/apache/iotdb/db/qp/physical/crud/InsertMultiTabletPlan.java

+  @Override
+  public void serializeToWAL(IWALByteBufferView buffer) {
+    int type = PhysicalPlanType.MULTI_BATCH_INSERT.ordinal();
+    buffer.put((byte) type);
+    buffer.putInt(insertTabletPlanList.size());
+    for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+      insertTabletPlan.subSerialize(buffer);
+    }
+
+    buffer.putInt(parentInsertTabletPlanIndexList.size());
+    for (Integer index : parentInsertTabletPlanIndexList) {
+      buffer.putInt(index);
+    }
+  }
+


Any differences with serializeImpl? Add some Javadoc?

The code logic are same, but IWALByteBufferView encapsulates some actions to deal with BufferOverflowException.

It'll be better to add some javadoc about it.

Is current javadoc in the IWALByteBufferView interface ok?

HTHou · 2022-03-28T09:50:45Z

server/src/main/java/org/apache/iotdb/db/qp/physical/crud/InsertPlan.java

 import java.util.Set;

-public abstract class InsertPlan extends PhysicalPlan {
+public abstract class InsertPlan extends PhysicalPlan implements WALEditValue {


Why implements WALEditValue?

Use WALEditValue to manage InsertPlan, DeletePlan and IMemTable, you can see more details in the WALEdit class.

# Conflicts: # node-commons/src/main/java/org/apache/iotdb/commons/conf/IoTDBConstant.java # server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java # server/src/main/java/org/apache/iotdb/db/engine/StorageEngine.java # server/src/main/java/org/apache/iotdb/db/utils/EnvironmentUtils.java # server/src/main/java/org/apache/iotdb/db/writelog/recover/LogReplayer.java # server/src/test/java/org/apache/iotdb/db/writelog/PerformanceTest.java # server/src/test/java/org/apache/iotdb/db/writelog/recover/DeviceStringTest.java # server/src/test/java/org/apache/iotdb/db/writelog/recover/LogReplayerTest.java # server/src/test/java/org/apache/iotdb/db/writelog/recover/RecoverResourceFromReaderTest.java # server/src/test/java/org/apache/iotdb/db/writelog/recover/SeqTsFileRecoverTest.java # server/src/test/java/org/apache/iotdb/db/writelog/recover/UnseqTsFileRecoverTest.java

server/src/assembly/resources/conf/iotdb-engine.properties

server/src/main/java/org/apache/iotdb/db/engine/memtable/AbstractMemTable.java

server/src/main/java/org/apache/iotdb/db/wal/utils/WALSubmitter.java

server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileProcessor.java

server/src/main/java/org/apache/iotdb/db/wal/buffer/WALEdit.java

server/src/main/java/org/apache/iotdb/db/wal/buffer/AbstractWALBuffer.java

server/src/main/java/org/apache/iotdb/db/wal/buffer/WALBuffer.java

# Conflicts: # server/src/main/java/org/apache/iotdb/db/utils/EnvironmentUtils.java

# Conflicts: # library-udf/src/test/java/org/apache/iotdb/library/anomaly/AnomalyTests.java # server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java # server/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java # server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileProcessor.java # server/src/main/java/org/apache/iotdb/db/writelog/node/ExclusiveWriteLogNode.java # server/src/main/java/org/apache/iotdb/db/writelog/node/WriteLogNode.java

…er_meta * remotes/upstream/master: [IOTDB-2872] Rename vsg to dataRegion (apache#5470) [IOTDB-2806][InfluxDB] Compatibility of Apache IoTDB with InfluxDB - Complete UserGuide (apache#5351) [IOTDB-2658] Generate logical plan for query statement —— UT & Raw Data Query & Aggregation Query (apache#5469) [IOTDB-1614] New WAL (apache#5320) Update ContributeGuide.md (apache#5463) user SerializeDeserializeUtil in confignode/AuthorPlan (apache#5467) Modify the jdbc query time column to be empty (apache#5281) [IOTDB-2827] Batch insert in new cluster (apache#5412) [IOTDB-2801] New storage engine framework (apache#5357) [IOTDB-2807]Speed up the cross space compaction by multi-threads (apache#5415) Fix coverage check [IOTDB-2864] Fix Read-only occurred when insert Text values to aligned timeseries (apache#5460) [IOTDB-2803]add AlterTimeSeriesNode and CreateAlignedTimeSeriesNode to PlanNodeType and its serialize and deserialize (apache#5444) [IOTDB-2841] add permission manager model (apache#5401) # Conflicts: # server/src/main/java/org/apache/iotdb/db/mpp/sql/analyze/Analyzer.java # server/src/main/java/org/apache/iotdb/db/mpp/sql/planner/DistributionPlanner.java # server/src/main/java/org/apache/iotdb/db/mpp/sql/planner/LogicalPlanner.java # server/src/main/java/org/apache/iotdb/db/mpp/sql/planner/plan/PlanFragment.java # server/src/main/java/org/apache/iotdb/db/mpp/sql/planner/plan/node/PlanNodeType.java # server/src/main/java/org/apache/iotdb/db/mpp/sql/planner/plan/node/metedata/read/DevicesMetaScanNode.java # server/src/main/java/org/apache/iotdb/db/mpp/sql/planner/plan/node/source/SeriesScanNode.java # server/src/main/java/org/apache/iotdb/db/mpp/sql/planner/plan/node/source/SourceNode.java # server/src/main/java/org/apache/iotdb/db/mpp/sql/statement/StatementVisitor.java # server/src/test/java/org/apache/iotdb/db/mpp/sql/plan/LogicalPlannerTest.java

HeimingZ added 14 commits March 20, 2022 13:34

add io

2fe230f

add buffer

6707d54

add checkpoint manager

f9228a5

add wal node

2636534

add wal manager

7f5ba02

add wal recover

fb671dc

integrate

5565d8c

fix some error

3eda5dd

remove old code

9ed4991

fix conflict

0a631a2

update license

74c768f

fix cluster

9ea8d37

fix restartDaemon

8f6121b

wangchao316 reviewed Mar 23, 2022

View reviewed changes

wangchao316 reviewed Mar 24, 2022

View reviewed changes

HeimingZ added 6 commits March 24, 2022 21:27

use async mode in integration test

a363272

fix test for async mode

b2286c2

Merge branch 'master' into new-wal

31ded21

improve wal buffer in async mode

9875198

modify annotations

2d1946c

rename parameters name

99a25e4

HTHou reviewed Mar 28, 2022

View reviewed changes

HeimingZ added 4 commits March 28, 2022 19:28

add memTable snapshot

0ae1761

rename schemaEngine to schemaProcessor

bc13cc4

add wal file size threshold parameter

0379d15

qiaojialin reviewed Mar 31, 2022

View reviewed changes

fix for review

4eff18a

HeimingZ added 17 commits March 31, 2022 20:26

remove fsync checkpoint thread

2de773c

fix recover

bf98dc6

recover all unsealed files

a259158

rename wal edit to wal entry

d72db6c

rename buffer to stream

70b8518

Merge branch 'master' into new-wal

8299cc7

fix concurrent error on start and end field of InsertTabletPlan

236ff51

fix infinite loop

936faf6

trigger memTable snapshot by wal file ttl

6fec389

remove timed close tsfile and restrict memTable snapshot number

a834d85

fix time partition recover

0c97c7f

Merge branch 'master' into new-wal

8a008af

# Conflicts: # server/src/main/java/org/apache/iotdb/db/utils/EnvironmentUtils.java

fix error of merging master

5b01753

fix timed delete outdated wal files

bd5f9c2

fix merge error

5edb6f1

remove invalid files

c092399

qiaojialin approved these changes Apr 10, 2022

View reviewed changes

JackieTien97 approved these changes Apr 10, 2022

View reviewed changes

qiaojialin merged commit 3c81f56 into apache:master Apr 10, 2022

HeimingZ deleted the new-wal branch April 22, 2022 07:21

Conversation

HeimingZ commented Mar 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wangchao316 left a comment

Choose a reason for hiding this comment

Uh oh!

HeimingZ commented Mar 23, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

HeimingZ commented Mar 22, 2022 •

edited

Loading