[IOTDB-173] add batch rpc interface #351

qiaojialin · 2019-08-22T08:24:12Z

write total 90000 points, batch size = 100, 300 batch

jdbc batch statement

batch rpc interface:

docs/Documentation/UserGuide/7-Client API/1-Client API.md

example/client/src/main/java/org/apache/iotdb/client/SessionTest.java

example/client/src/main/java/org/apache/iotdb/client/StatementTest.java

service-rpc/src/main/thrift/rpc.thrift

server/src/main/java/org/apache/iotdb/db/service/TSServiceImpl.java

jixuan1989 · 2019-08-22T11:21:37Z

server/src/main/java/org/apache/iotdb/db/service/TSServiceImpl.java

+      for (Integer result : results) {
+        long t2 = System.currentTimeMillis();
+        isAllSuccessful = isAllSuccessful && (result == TS_StatusCode.SUCCESS_STATUS.getValue() ? true : false);
+        Measurement.INSTANCE.addOperationLatency(Operation.EXECUTE_ONE_SQL_IN_BATCH, t2);


t1 is the whole batch, t2 is each statement

Try using more meaningful names.

if so, I suggest remove this ..... it is almost 0...

jixuan1989 · 2019-08-22T12:46:53Z

server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileProcessor.java

+      try {
+        getLogNode().write(batchInsertPlan);
+      } catch (IOException e) {
+        logger.error("write WAL failed", e);


change values of results

jixuan1989 · 2019-08-22T12:54:41Z

server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java

+      latestTimeForEachDevice.putIfAbsent(batchInsertPlan.getDeviceId(), Long.MIN_VALUE);
+      latestFlushedTimeForEachDevice.putIfAbsent(batchInsertPlan.getDeviceId(), Long.MIN_VALUE);
+
+      List<Integer> results = new ArrayList<>();


use Integer[] instead

jixuan1989 · 2019-08-22T12:55:06Z

server/src/main/java/org/apache/iotdb/db/engine/memtable/WritableMemChunk.java

    switch (dataType) {
      case BOOLEAN:
-        putBoolean(insertTime, (Boolean)value);
+        //TODO if indexes.size() == times.length, could we write a chunk at a time


implement this. It is important!

client/src/main/java/org/apache/iotdb/session/IoTDBRowBatch.java

client/src/main/java/org/apache/iotdb/session/Session.java

server/src/main/java/org/apache/iotdb/db/engine/StorageEngine.java

server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileProcessor.java

server/src/main/java/org/apache/iotdb/db/qp/executor/QueryProcessExecutor.java

server/src/main/java/org/apache/iotdb/db/qp/physical/crud/BatchInsertPlan.java

jt2594838 · 2019-08-23T02:23:06Z

server/src/main/java/org/apache/iotdb/db/service/TSServiceImpl.java

+      for (Integer result : results) {
+        long t2 = System.currentTimeMillis();
+        isAllSuccessful = isAllSuccessful && (result == TS_StatusCode.SUCCESS_STATUS.getValue() ? true : false);
+        Measurement.INSTANCE.addOperationLatency(Operation.EXECUTE_ONE_SQL_IN_BATCH, t2);


Try using more meaningful names.

client/pom.xml

example/client/src/main/java/org/apache/iotdb/client/ClientExample.java

server/src/main/java/org/apache/iotdb/db/utils/MemUtils.java

jt2594838

Looks fine.

…sword

jixuan1989 · 2019-08-25T15:10:29Z

I have tested the interface and when using this interface to write data in batch
e.g., write 1000 data points of device 1 first, then 1000 data points of device 2, then ...
Comparing with using Statement.executeBatch(), using this interface is 5 faster than Statement.executeBatch().

jt2594838 · 2019-08-26T06:22:21Z

example/session/src/main/java/org/apache/iotdb/SessionExample.java

+  public static void main(String[] args) throws ClassNotFoundException, IoTDBSessionException {
+    Class.forName("org.apache.iotdb.jdbc.IoTDBDriver");
+    try (Connection connection = DriverManager.getConnection("jdbc:iotdb://127.0.0.1:6667/", "root", "root");
+        Statement statement = connection.createStatement()) {


Actually, I hope you can add those metadata interfaces so that you may avoid using jdbc in your rpc example.

Actually, I hope that too, I will add them in the next PR.

qiaojialin added 4 commits August 22, 2019 15:20

add batch rpc interface

ceaa9b1

fix write too many times bug

e31d4f5

merge master to resolve conflict

f0fb8c2

fix columns to rows

4281082

jixuan1989 reviewed Aug 22, 2019

View reviewed changes

qiaojialin added 3 commits August 22, 2019 18:45

add license and change private to public

f9a1ab8

fix reviews

a098310

add license in BatchInsertPlan

22bab90

jixuan1989 reviewed Aug 22, 2019

View reviewed changes

jt2594838 requested changes Aug 23, 2019

View reviewed changes

jt2594838 and others added 4 commits August 23, 2019 13:40

add array-parametered interfaces

32902bf

change data value list in rpc to bytes

eb0935f

Merge remote-tracking branch 'origin/rpc_client' into rpc_client

9a872e7

change time-value in rpc to bytes

965e72d

qiaojialin changed the title ~~add batch rpc interface~~ [IOTDB-173] add batch rpc interface Aug 23, 2019

qiaojialin added 3 commits August 23, 2019 19:44

fix test bug

69f211f

rename Client to Cli and Session to Client

e7ea34d

fix when batch is not full, time serialize overflow buffer size bug

895c9ba

jt2594838 requested changes Aug 23, 2019

View reviewed changes

client/pom.xml Outdated Show resolved Hide resolved

example/client/src/main/java/org/apache/iotdb/client/ClientExample.java Outdated Show resolved Hide resolved

server/src/main/java/org/apache/iotdb/db/utils/MemUtils.java Outdated Show resolved Hide resolved

jt2594838 reviewed Aug 23, 2019

View reviewed changes

remove IoTDBDataType in rpc, use int32 instead

c9d040c

qiaojialin force-pushed the rpc_client branch from 0e493c7 to c9d040c Compare August 23, 2019 15:01

qiaojialin and others added 7 commits August 24, 2019 11:25

add session module and rename cli to client

5b1a318

fix windows echo cli to client

0777569

make session module depend on service-rpc instead of jdbc

ca45800

fix putsInts in IntTVList

553055d

add license

6320a45

fix IT test, IoTDBRPCException -> IoTDBSQLException, default user/pas…

dd879eb

…sword

fix session example caught exception

7714ec4

jixuan1989 approved these changes Aug 25, 2019

View reviewed changes

jt2594838 approved these changes Aug 26, 2019

View reviewed changes

qiaojialin merged commit 6ce1f41 into master Aug 26, 2019

qiaojialin deleted the rpc_client branch September 2, 2019 12:33

neuyilan pushed a commit that referenced this pull request Feb 28, 2023

[To 1.0 ] Support Sync Data to AP (#351)

f21e1ce

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[IOTDB-173] add batch rpc interface #351

[IOTDB-173] add batch rpc interface #351

qiaojialin commented Aug 22, 2019 •

edited

jixuan1989 Aug 22, 2019

qiaojialin Aug 23, 2019

jt2594838 Aug 23, 2019

jixuan1989 Aug 23, 2019

jixuan1989 Aug 22, 2019

qiaojialin Aug 23, 2019

jixuan1989 Aug 22, 2019

qiaojialin Aug 23, 2019

jixuan1989 Aug 22, 2019

qiaojialin Aug 23, 2019

jt2594838 Aug 23, 2019

jt2594838 left a comment

jixuan1989 commented Aug 25, 2019

jt2594838 Aug 26, 2019

qiaojialin Aug 26, 2019

[IOTDB-173] add batch rpc interface #351

[IOTDB-173] add batch rpc interface #351

Conversation

qiaojialin commented Aug 22, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jt2594838 left a comment

Choose a reason for hiding this comment

jixuan1989 commented Aug 25, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiaojialin commented Aug 22, 2019 •

edited