[IOTDB-68] New shared-nothing cluster #460

qiaojialin · 2019-10-18T09:37:48Z

No description provided.

jixuan1989 · 2020-01-02T16:41:30Z

RaftMember.java
Now we have to send request to all nodes and wait for quorum nodes responses. then we can commit the log.

How about:

guarantee there is only one connection between two nodes.
The leader can send the second req to a node after the first req is sent to the node.
However, the leader can commit log index i until i-1 is committed.

jixuan1989 · 2020-01-03T08:01:39Z

PartitionGroup extends ArrayList is a little heavy..
an array is enough.

sonarcloud · 2020-02-07T15:24:29Z

SonarCloud Quality Gate failed.

28 Bugs
11 Vulnerabilities (and 4 Security Hotspots to review)
200 Code Smells

No Coverage information
0.3% Duplication

fix some regression bugs

fix conf bug

[IOTDB-884] group createMultiTimeseriesPlan by partitionGroup

fix takeSnapshot bug

# Conflicts: # docs/UserGuide/Server/Cluster Setup.md # docs/UserGuide/System Tools/NodeTool.md # docs/zh/UserGuide/Server/Cluster Setup.md # docs/zh/UserGuide/System Tools/NodeTool.md # server/src/main/java/org/apache/iotdb/db/engine/StorageEngine.java # server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java # server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileProcessor.java # server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileResource.java # server/src/main/java/org/apache/iotdb/db/metadata/MManager.java # server/src/main/java/org/apache/iotdb/db/metadata/MTree.java # server/src/main/java/org/apache/iotdb/db/metadata/PartialPath.java # server/src/main/java/org/apache/iotdb/db/qp/executor/PlanExecutor.java # server/src/main/java/org/apache/iotdb/db/qp/physical/PhysicalPlan.java # server/src/main/java/org/apache/iotdb/db/qp/physical/crud/InsertPlan.java # server/src/main/java/org/apache/iotdb/db/qp/physical/crud/InsertRowPlan.java # server/src/main/java/org/apache/iotdb/db/qp/physical/crud/InsertTabletPlan.java # server/src/main/java/org/apache/iotdb/db/service/TSServiceImpl.java # server/src/main/java/org/apache/iotdb/db/utils/CommonUtils.java # server/src/main/java/org/apache/iotdb/db/utils/SchemaUtils.java # server/src/test/java/org/apache/iotdb/db/metadata/MManagerBasicTest.java # server/src/test/java/org/apache/iotdb/db/sync/receiver/load/FileLoaderTest.java # thrift/src/main/thrift/cluster.thrift

fix handling when client is null trigger election when leader is down

# Conflicts: # server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java

fix maybeTerm

jt2594838 · 2020-12-04T07:14:29Z

The remaining code smells are duplicated code blocks caused by single inheritence in Java, and there are currently no elegant way to fix them.

cluster/src/main/java/org/apache/iotdb/cluster/server/member/RaftMember.java

...er/src/main/java/org/apache/iotdb/cluster/server/handlers/caller/AppendNodeEntryHandler.java

server/src/main/java/org/apache/iotdb/db/engine/StorageEngine.java

jixuan1989 · 2020-12-04T17:45:46Z

server/src/main/java/org/apache/iotdb/db/engine/flush/MemTableFlushTask.java

@@ -235,6 +235,8 @@ public void run() {
            ChunkWriterImpl chunkWriter = (ChunkWriterImpl) ioMessage;
            chunkWriter.writeToFileWriter(this.writer);
          } else {
+            this.writer.setMinPlanIndex(memTable.getMinPlanIndex());


why add min/max planIndex on ChunkGroupFooter?

It mainly used for a follower to catch up to the leader when doing snapshot task to filter tsfiles, for details please see https://issues.apache.org/jira/browse/IOTDB-918

jixuan1989 · 2020-12-04T18:52:38Z

tsfile/src/main/java/org/apache/iotdb/tsfile/read/reader/page/PageReader.java

@@ -162,7 +163,11 @@ public Statistics getStatistics() {

  @Override
  public void setFilter(Filter filter) {
-    this.filter = filter;
+    if (this.filter == null) {


@JackieTien97 check whether this has some negative effect.

Actually, i think we should just use the filter to overwrite the former one. The former one is actually a time filter and it has been checked when the PageReader is constructed.

And alse don't forget to change back the MemPageReader

I previously found that data of this page will miss the previous filter, meaning some data are not time-filtered. We may remove it later if you are confident.

jixuan1989 · 2020-12-04T18:53:46Z

tsfile/format-changelist.md

@@ -19,6 +19,11 @@

 -->

+# 0.10.x (version-2) -> 0.12.x (version-3)
+| PR#   | Name                                                        | Author          | Changes                                                      |


need to say why on somewhere.

[Distributed] Make cluster configuration consensus with the single-node

JackieTien97 · 2020-12-05T11:10:09Z

jdbc/src/main/java/org/apache/iotdb/jdbc/IoTDBStatement.java

+   * <p>
+   * (1) return IoTDBJDBCResultSet or IoTDBNonAlignJDBCResultSet (2) simply get executed


format back

In this way, the format will not be modified when the shortcut keys are used in the idea to automatically format, otherwise, the format will be modified if the shortcut keys are used.

JackieTien97 · 2020-12-05T11:15:21Z

server/src/assembly/resources/conf/logback.xml

@@ -52,7 +52,7 @@
            <maxIndex>10</maxIndex>
        </rollingPolicy>
        <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
-            <maxFileSize>1MB</maxFileSize>
+            <maxFileSize>10MB</maxFileSize>


Why we need to change the default size of logs

it's ok we use the server‘s logback's configuration as default configuration

JackieTien97 · 2020-12-05T11:50:07Z

server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java

+          fileProcessorMap.remove(timePartition);
+        }
+      } catch (Exception e) {
+        logger.error("Cannot close {}", tsFileResource, e);


Should we rethrow the exception and add a finally block to do the remove things

The exception will also only be logged in the upper level, so it is fine to do the logging here. When there is any exception, removing it may not be safe anymore, we'd better leave it be.

JackieTien97 · 2020-12-05T12:02:22Z

server/src/main/java/org/apache/iotdb/db/engine/compaction/no/NoCompactionTsFileManagement.java

@@ -56,7 +56,7 @@ public NoCompactionTsFileManagement(String storageGroupName, String storageGroup
    if (sequence) {
      return new ArrayList<>(sequenceFileTreeSet);
    } else {
-      return unSequenceFileList;
+      return new ArrayList<>(unSequenceFileList);


No need to copy it again here, it won't be changed outside.
Actually it's better that the interface's return type should be changed to Collection

But it may be changed indide, making an iteration throws a ConcurrentModificationException.

JackieTien97 · 2020-12-05T12:16:44Z

server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileResource.java

+      ReadWriteIOUtils.write(maxPlanIndex, outputStream);
+      ReadWriteIOUtils.write(minPlanIndex, outputStream);


duplicated with the following lines

JackieTien97 · 2020-12-05T12:23:06Z

server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileResource.java

+    try {
+      Files.deleteIfExists(file.toPath());
+    } catch (IOException e) {
+      logger.warn("TsFile {} cannot be deleted: {}", file, e.getMessage());


better to rethrow the exception

This will break the code structure of many callers, and they can do nothing more than logging it.

JackieTien97 · 2020-12-05T12:23:22Z

server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileResource.java

+      Files.deleteIfExists(
+          fsFactory.getFile(file.getPath() + ModificationFile.FILE_SUFFIX).toPath());
+    } catch (IOException e) {
+      logger.warn("ModificationFile {} cannot be deleted: {}", file, e.getMessage());


same as above

JackieTien97 · 2020-12-05T12:23:30Z

server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileResource.java

+    try {
+      Files.deleteIfExists(fsFactory.getFile(file.getPath() + RESOURCE_SUFFIX).toPath());
+    } catch (IOException e) {
+      logger.warn("TsFileResource {} cannot be deleted: {}", file, e.getMessage());


same as above

JackieTien97 · 2020-12-05T12:42:55Z

server/src/main/java/org/apache/iotdb/db/metadata/PartialPath.java

+  private String measurementAlias = "";
  // alias of time series used in SELECT AS
-  private String tsAlias = null;
+  private String tsAlias = "";


How do we distinguish whether the measurement has an empty string as its alias or it doesn't have alias?

@Alima777 PTAL(do you think it's a problem?)

as far as I know, iotdb does not support empty string as an alias, so it's ok to use an empty string here.

We do not support empty strings within timeseries paths, so there is even less meaning to support it in alias, so both make no difference.

From my perspective, it's fine.

JackieTien97 · 2020-12-05T13:04:14Z

tsfile/src/main/java/org/apache/iotdb/tsfile/read/reader/page/PageReader.java

@@ -162,7 +163,11 @@ public Statistics getStatistics() {

  @Override
  public void setFilter(Filter filter) {
-    this.filter = filter;
+    if (this.filter == null) {


Actually, i think we should just use the filter to overwrite the former one. The former one is actually a time filter and it has been checked when the PageReader is constructed.

And alse don't forget to change back the MemPageReader

jixuan1989 · 2020-12-05T13:19:54Z

service-rpc/src/main/java/org/apache/iotdb/rpc/NonOpenTransport.java

+  }
+
+  @Override
+  public boolean isOpen() { return true; }


how about set to false once close() is called.

No point. close() means nothing to such classes. You may find similar codes in other thrift Transports like MemTransport.

server/src/test/java/org/apache/iotdb/db/writelog/recover/SeqTsFileRecoverTest.java

server/src/test/java/org/apache/iotdb/db/writelog/WriteLogNodeTest.java

server/src/main/java/org/apache/iotdb/db/tools/TsFileResourcePrinter.java

server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileProcessor.java

server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java

# Conflicts: # server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java

# Conflicts: # server/src/main/java/org/apache/iotdb/db/sync/sender/transfer/SyncClient.java

sonarcloud · 2020-12-07T14:16:44Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
11 Code Smells

No Coverage information
0.2% Duplication

The version of Java (1.8.0_252) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 11.
Read more here

jixuan1989 · 2020-12-08T15:20:53Z

Though there are many works left to do for this issue, I have to say you have reached a good milestone. So, I will merge this PR tonight.
Nice work, guys!

Co-authored-by: HouliangQi <neuyilan@163.com> Co-authored-by: wangyanhong <1101967410@qq.com> Co-authored-by: chaow <runhuster@foxmail.com> Co-authored-by: Jiang Tian <jt2594838@163.com> Co-authored-by: chaow <xunmengzhuiyi@gmail.com> Co-authored-by: LebronAl <TXYPotato@gmail.com> Co-authored-by: Ring-k <yuyuankang@hotmail.com> Co-authored-by: xiangdong huang <sainthxd@gmail.com>

qiaojialin mentioned this pull request Feb 23, 2020

[IOTDB-68]Add basic cluster framework, non-query module and query metadata module #178

Closed

qiaojialin changed the title ~~New shared-nothing cluster~~ [IOTDB-68] New shared-nothing cluster Feb 23, 2020

neuyilan and others added 25 commits October 27, 2020 20:26

add tests

7caca25

add use persist log to snapshot

851ef63

fix conf bug

e3ccf65

remove useless lines

40b08e1

fix fristLogIndex bug

7949b1d

add switch of whether use persist logs on disk to catchup

6b83b80

fix log manager print messags

106afb2

Merge pull request #1866 from apache/kyy

5c76102

fix some regression bugs

Merge pull request #1877 from yhwangBonc/cluster_new_fix_conf_bug

bf25c71

fix conf bug

Merge pull request #1854 from mychaow/cluster_new

2bade4f

[IOTDB-884] group createMultiTimeseriesPlan by partitionGroup

Merge pull request #1872 from mychaow/snapshotfix

00bd239

fix takeSnapshot bug

add getLogs UT

b66f25d

change forceRaftLogPeriodInMS to 1000

8779750

use persist log when getTerm not found in memory

482e36b

fix test

586c921

remove ClusterFlushPolicy

e4eb46b

limit log size of logDispatcher

35fa060

fix log level

70550c1

add recovery log data file

29de7ef

fix bug

71a20fb

fix snapshot overlaps logs during snapshot

0aad6a1

fix match index not updated after snapshot catch-up

d53ebb3

revert the test log level to warn

bb5ba16

fix error message when node is down

4f1a193

fix handling when client is null trigger election when leader is down

jt2594838 added 8 commits December 3, 2020 17:43

refine timer

6c033f6

accelerate handling when the quorum fails

d1b9e45

Merge branch 'master' into cluster_new

7a088ff

# Conflicts: # server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java

update cluster denpendency

b3c2ce3

fix tests

9e33a73

fix code smells

f409921

fix maybeTerm

19d57cf

fix isFileAlreadyExist

39cb51a

fix maybeTerm

remove duplicated configuration files

c228993

jixuan1989 requested changes Dec 4, 2020

View reviewed changes

jixuan1989 reviewed Dec 4, 2020

View reviewed changes

Merge pull request #2189 from apache/cluster_configuration_consensus

0736499

[Distributed] Make cluster configuration consensus with the single-node

JackieTien97 requested changes Dec 5, 2020

View reviewed changes

jixuan1989 reviewed Dec 5, 2020

View reviewed changes

server/src/test/java/org/apache/iotdb/db/writelog/recover/SeqTsFileRecoverTest.java Show resolved Hide resolved

jixuan1989 reviewed Dec 5, 2020

View reviewed changes

server/src/test/java/org/apache/iotdb/db/writelog/WriteLogNodeTest.java Show resolved Hide resolved

jixuan1989 reviewed Dec 5, 2020

View reviewed changes

server/src/main/java/org/apache/iotdb/db/tools/TsFileResourcePrinter.java Outdated Show resolved Hide resolved

jixuan1989 reviewed Dec 7, 2020

View reviewed changes

server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileProcessor.java Show resolved Hide resolved

jixuan1989 reviewed Dec 7, 2020

View reviewed changes

server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java Show resolved Hide resolved

jt2594838 added 4 commits December 7, 2020 18:38

Merge branch 'master' into cluster_new

c311192

# Conflicts: # server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java

fix reviews

ff06133

fix reviews

2055427

Merge branch 'master' into cluster_new

af73cd6

# Conflicts: # server/src/main/java/org/apache/iotdb/db/sync/sender/transfer/SyncClient.java

jixuan1989 merged commit 580b012 into master Dec 8, 2020

qiaojialin deleted the cluster_new branch June 16, 2021 07:25

		* <p>
		* (1) return IoTDBJDBCResultSet or IoTDBNonAlignJDBCResultSet (2) simply get executed

		ReadWriteIOUtils.write(maxPlanIndex, outputStream);
		ReadWriteIOUtils.write(minPlanIndex, outputStream);

[IOTDB-68] New shared-nothing cluster #460

[IOTDB-68] New shared-nothing cluster #460

Conversation

qiaojialin commented Oct 18, 2019

jixuan1989 commented Jan 2, 2020

jixuan1989 commented Jan 3, 2020

sonarcloud bot commented Feb 7, 2020

jt2594838 commented Dec 4, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JackieTien97 Dec 5, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JackieTien97 Dec 5, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented Dec 7, 2020

jixuan1989 commented Dec 8, 2020

JackieTien97 Dec 5, 2020 •

edited

JackieTien97 Dec 5, 2020 •

edited