[IOTDB-915]Add raft log persist mechanism and use persist log to catch up #1865

neuyilan · 2020-10-26T11:34:21Z

No description provided.

OneSizeFitsQuorum

Great Work!

cluster/src/assembly/resources/conf/iotdb-cluster.properties

cluster/src/main/java/org/apache/iotdb/cluster/config/ClusterConfig.java

OneSizeFitsQuorum · 2020-10-30T03:24:11Z

cluster/src/main/java/org/apache/iotdb/cluster/log/manage/RaftLogManager.java

@@ -292,15 +292,29 @@ public long getLastLogIndex() {
  public long getTerm(long index) throws EntryCompactedException {
    long dummyIndex = getFirstIndex() - 1;
    if (index < dummyIndex) {
+      // search in disk


I thought these code should be in the function maybeTerm in committedEntryManager, which should manager all persisted entries, it will throw a EntryCompactedException or get log from disk if isEnableRaftLogPersistence is enabled.
BTW, I'm a little confused about whether we should get log from disk in function getTerm,If so, then maybe we should change isEnableUsePersistLogOnDiskToCatchUp's name

if we manage all the persisted entries(committed or persisted) on the committedEntryManager, it's better to move the code in committedEntryManager. but now, the committedEntryManager only manager the commit log in memory. and the LogManager managers the UnCommittedEntryManager CommittedEntryManager and StableEntryManager, I think CommittedEntryManager is as logs in memory, StableEntryManager is as logs in the disk, so I think it's ok to leave the code here, the log manager knows where to get the logs

besides, I argue with you that, when get term, we should not care the isEnableUsePersistLogOnDiskToCatchUp parpermeter, if isEnableRaftLogPersistence is enabled is ok

OK.As the logmanager has been given more functionality, I will try to merge UnCommittedEntryManager and CommittedEntryManager to a MemoryLogmanager to avoid redundant entry shift laterly.
I'm OK with here~

...r/src/main/java/org/apache/iotdb/cluster/log/manage/serializable/SyncLogDequeSerializer.java

cluster/src/assembly/resources/conf/iotdb-cluster.properties

...r/src/main/java/org/apache/iotdb/cluster/log/manage/serializable/SyncLogDequeSerializer.java

OneSizeFitsQuorum · 2020-10-30T09:40:25Z

...r/src/main/java/org/apache/iotdb/cluster/log/manage/serializable/SyncLogDequeSerializer.java

      .allocate(ClusterDescriptor.getInstance().getConfig().getRaftLogBufferSize());
+  private ByteBuffer logIndexBuffer = ByteBuffer


As you have used logIndexOffsetList to record offset in putLog, I doubt whether it's necessary to maintain this buffer in putLog.Maybe You can generate index buffer in flushLogBuffer .

I designed it on purpose. If you set a temporary buffer, GC will occur. I think it is better to set a buffer that is not GC?

OneSizeFitsQuorum · 2020-10-30T09:52:52Z

...r/src/main/java/org/apache/iotdb/cluster/log/manage/serializable/SyncLogDequeSerializer.java

+   * the offset of the log's index, for example, the first value is the offset of index
+   * ${firstLogIndex}, the second value is the offset of index ${firstLogIndex+1}
+   */
+  private List<Long> logIndexOffsetList;


As currently the default maxSize of each data file is 1G,it's seems a Integer is able to record the offset.

sure, but the parameter of the max size of the log data file is left to the user to set, so if the user config larger than an integer, it would occur errors, or we could limit the range of user configuration for the maxSize parameter.

OK, Just let it go~Long is OK for me ~

jt2594838

Great, only minor problems.

cluster/src/main/java/org/apache/iotdb/cluster/log/catchup/LogCatchUpTask.java

...r/src/main/java/org/apache/iotdb/cluster/log/manage/serializable/SyncLogDequeSerializer.java

jt2594838 · 2020-10-30T09:42:46Z

...r/src/main/java/org/apache/iotdb/cluster/log/manage/serializable/SyncLogDequeSerializer.java

-      minAvailableVersion = getFileVersion(logFile);
-      serializeMeta(meta);
+  @Override
+  public List<Log> getLogs(long startIndex, long endIndex) {


I think we should enforce a limit on this method to avoid out ot memory when the range is too long.

I add one parameter to limit the max number of logs per fetch

neuyilan · 2020-11-01T03:09:55Z

#1865 (comment)

the log whose index large than the applied index is all thought has been applied, the wal may make sure when re start iotdb to redo the logs.

jt2594838 · 2020-11-03T01:40:44Z

It seems we got compilation error.

…og_catch_up

neuyilan marked this pull request as draft October 26, 2020 11:34

HTHou added the Module - Cluster PRs for the cluster module label Oct 26, 2020

neuyilan added 3 commits October 27, 2020 20:26

add raft log mechanism and use persist log to catch up

3382d6b

add test

b7cf81a

add tests

7caca25

neuyilan force-pushed the apache_cluster_new_1023_raft_log_catch_up branch from f075e52 to 7caca25 Compare October 27, 2020 12:28

neuyilan added 2 commits October 27, 2020 20:58

add use persist log to snapshot

851ef63

remove useless lines

40b08e1

neuyilan marked this pull request as ready for review October 28, 2020 01:42

neuyilan marked this pull request as draft October 28, 2020 01:55

neuyilan marked this pull request as ready for review October 28, 2020 02:29

neuyilan marked this pull request as draft October 28, 2020 02:48

neuyilan added 7 commits October 28, 2020 11:50

fix fristLogIndex bug

7949b1d

add switch of whether use persist logs on disk to catchup

6b83b80

fix log manager print messags

106afb2

add getLogs UT

b66f25d

change forceRaftLogPeriodInMS to 1000

8779750

use persist log when getTerm not found in memory

482e36b

add recovery log data file

29de7ef

neuyilan changed the title ~~[IOTDB-915]Add raft log mechanism and use persist log to catch up~~ [IOTDB-915]Add raft log persist mechanism and use persist log to catch up Oct 29, 2020

neuyilan marked this pull request as ready for review October 29, 2020 08:21

neuyilan added 4 commits October 29, 2020 16:24

revert the test log level to warn

bb5ba16

fix get log term

2463c42

fix ut on windows

30542c1

fix ut

9eca1bc

OneSizeFitsQuorum reviewed Oct 30, 2020

View reviewed changes

jt2594838 requested changes Oct 30, 2020

View reviewed changes

neuyilan added 2 commits October 30, 2020 19:26

fix windows ut

bc72be1

fix revoew

fa7f863

neuyilan added 5 commits November 1, 2020 12:17

fix review and add per fetch limit when get logs from disks

e8a950a

fix ci

6e38d4b

fix review

b6e5983

try to fix ci

70f8abc

try to fix windows ci

777d8d7

jt2594838 approved these changes Nov 2, 2020

View reviewed changes

neuyilan added 3 commits November 2, 2020 15:02

fix exception not catch

69561c2

fix null exception

6de89e2

resolve confilict

6711945

neuyilan added 3 commits November 3, 2020 10:05

fix IT bug

d19f3b7

Merge branch 'apache_cluster_new' into apache_cluster_new_1023_raft_l…

abf53f6

…og_catch_up

fix IT win bug

c745099

jt2594838 merged commit 4715ad5 into apache:cluster_new Nov 3, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[IOTDB-915]Add raft log persist mechanism and use persist log to catch up #1865

[IOTDB-915]Add raft log persist mechanism and use persist log to catch up #1865

neuyilan commented Oct 26, 2020

OneSizeFitsQuorum left a comment

OneSizeFitsQuorum Oct 30, 2020

neuyilan Nov 1, 2020

OneSizeFitsQuorum Nov 1, 2020 •

edited

Loading

OneSizeFitsQuorum Oct 30, 2020 •

edited

Loading

neuyilan Nov 1, 2020

OneSizeFitsQuorum Nov 1, 2020

OneSizeFitsQuorum Oct 30, 2020

neuyilan Oct 30, 2020

OneSizeFitsQuorum Oct 31, 2020 •

edited

Loading

jt2594838 left a comment

jt2594838 Oct 30, 2020

neuyilan Nov 2, 2020

neuyilan commented Nov 1, 2020

jt2594838 commented Nov 3, 2020

		.allocate(ClusterDescriptor.getInstance().getConfig().getRaftLogBufferSize());
		private ByteBuffer logIndexBuffer = ByteBuffer

[IOTDB-915]Add raft log persist mechanism and use persist log to catch up #1865

[IOTDB-915]Add raft log persist mechanism and use persist log to catch up #1865

Conversation

neuyilan commented Oct 26, 2020

OneSizeFitsQuorum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

OneSizeFitsQuorum Nov 1, 2020 • edited Loading

Choose a reason for hiding this comment

OneSizeFitsQuorum Oct 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

OneSizeFitsQuorum Oct 31, 2020 • edited Loading

Choose a reason for hiding this comment

jt2594838 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neuyilan commented Nov 1, 2020

jt2594838 commented Nov 3, 2020

OneSizeFitsQuorum Nov 1, 2020 •

edited

Loading

OneSizeFitsQuorum Oct 30, 2020 •

edited

Loading

OneSizeFitsQuorum Oct 31, 2020 •

edited

Loading