[IOTDB-749] Handle select * from root OOM #1367

JackieTien97 · 2020-06-15T13:06:16Z

No description provided.

qiaojialin · 2020-06-17T07:42:49Z

server/src/assembly/resources/conf/iotdb-engine.properties

@@ -226,6 +226,10 @@ write_read_free_memory_proportion=6:3:1
 # primitive array size (length of each array) in array pool
 primitive_array_size=128

+# allowed max numbers of deduplicated path in one query
+# it's just an advised value, the real limitation will be the smaller one between this and the one we calculated
+max_deduplicated_path_num=10000


reduce to 1000?

server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java

server/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java

sonarcloud · 2020-06-29T08:07:46Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities (and 0 Security Hotspots to review)
3 Code Smells

No Coverage information
0.0% Duplication

The version of Java (1.8.0_252) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11.
Read more here

jixuan1989 · 2020-06-29T08:23:01Z

server/src/main/java/org/apache/iotdb/db/query/control/QueryResourceManager.java

+        if (totalFreeMemoryForRead.addAndGet(-estimatedMemoryUsage) >= 0) {
+          queryIdEstimatedMemoryMap.put(queryId, estimatedMemoryUsage);
+        } else {
+          totalFreeMemoryForRead.addAndGet(estimatedMemoryUsage);


why add memory here..

jixuan1989 · 2020-06-29T08:23:29Z

server/src/main/java/org/apache/iotdb/db/query/control/QueryResourceManager.java

+  private final AtomicLong totalFreeMemoryForRead;
+
+  // estimated size for one point memory size, the unit is byte
+  private static final long POINT_ESTIMATED_SIZE = 16L;


only works for non-string..

jixuan1989 · 2020-06-29T08:29:15Z

I register 10_000_000 timeseries by:

  public static void main(String[] args) throws ClassNotFoundException, SQLException {
    Class.forName("org.apache.iotdb.jdbc.IoTDBDriver");
    try (Connection connection = DriverManager
        .getConnection("jdbc:iotdb://127.0.0.1:6667/", "root", "root");
        Statement statement = connection.createStatement()) {
      int total = 10_000_000;
      try {
        statement.execute("SET STORAGE GROUP TO root.sg1");
        for (int i = 0; i < total; i++) {
          statement.execute(
              "CREATE TIMESERIES root.sg1.d1.s" + i
                  + " WITH DATATYPE=INT64, ENCODING=RLE, COMPRESSOR=SNAPPY");
          if (i % 10_000 == 0) {
            System.out.println((double) i / total * 100 + "%");
          }
        }
      } catch (IoTDBSQLException e) {
        System.out.println(e.getMessage());
      }
      
    } catch (IoTDBSQLException e) {
      System.out.println(e.getMessage());
    }
  }

And then call select * from root;
The CLI hangs..

SilverNarcissus

Look good to me~

SilverNarcissus · 2020-07-08T06:14:46Z

server/src/assembly/resources/conf/iotdb-engine.properties

@@ -233,6 +233,10 @@ write_read_free_memory_proportion=6:3:1
 # primitive array size (length of each array) in array pool
 primitive_array_size=128

+# allowed max numbers of deduplicated path in one query
+# it's just an advised value, the real limitation will be the smaller one between this and the one we calculated
+max_deduplicated_path_num=10000


From my perspective, 1000 is better. XD

neuyilan · 2020-10-14T04:27:44Z

server/src/assembly/resources/conf/iotdb-engine.properties

@@ -323,7 +327,7 @@ merge_throughput_mb_per_sec=16
 meta_data_cache_enable=true
 # Read memory Allocation Ratio: ChunkMetadataCache, ChunkCache, TimeSeriesMetadataCache and Free Memory Used in Query.
 # The parameter form is a:b:c:d, where a, b, c and d are integers. for example: 1:1:1:1 , 6:10:5:15
-chunkmeta_chunk_timeseriesmeta_free_memory_proportion=1:1:1:7
+chunkmeta_chunk_timeseriesmeta_free_memory_proportion=1:1:1:3:4


the value has 5 parameters, maybe the annotation and the attribute need to change?

I agree with that. The comments need to be modified.

Alima777

There are conflicts now, please fix it.

Alima777

LGTM. Please fix some small issues.

Alima777 · 2020-10-20T01:50:54Z

server/src/assembly/resources/conf/iotdb-engine.properties

@@ -323,7 +327,7 @@ merge_throughput_mb_per_sec=16
 meta_data_cache_enable=true
 # Read memory Allocation Ratio: ChunkMetadataCache, ChunkCache, TimeSeriesMetadataCache and Free Memory Used in Query.
 # The parameter form is a:b:c:d, where a, b, c and d are integers. for example: 1:1:1:1 , 6:10:5:15
-chunkmeta_chunk_timeseriesmeta_free_memory_proportion=1:1:1:7
+chunkmeta_chunk_timeseriesmeta_free_memory_proportion=1:1:1:3:4


I agree with that. The comments need to be modified.

Alima777 · 2020-10-20T02:23:07Z

server/src/main/java/org/apache/iotdb/db/qp/strategy/optimizer/ConcatPathOptimizer.java

  /**
   * Make 'SLIMIT&SOFFSET' take effect by trimming the suffixList and aggregations of the
   * selectOperator.
   *
-   * @param seriesLimit  is ensured to be positive integer
   * @param seriesOffset is ensured to be non-negative integer
   */
-  private void slimitTrim(SelectOperator select, int seriesLimit, int seriesOffset)
+  private void slimitTrim(SelectOperator select, int seriesOffset)
      throws LogicalOptimizeException {


From my perspective, limit is done in concatselect() while this method is only for trimming the offset of aggregations.
If I'm right, please modify the method name and java doc of it.

SteveYurongSu · 2020-10-21T06:09:41Z

server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java

  /**
   * Is dynamic parameter adapter enable.
   */
-  private boolean enableParameterAdapter = true;
+  private boolean enableParameterAdapter = false;


Why set it to false?

SteveYurongSu · 2020-10-21T06:59:00Z

server/src/main/java/org/apache/iotdb/db/metadata/MTree.java

@@ -777,18 +777,26 @@ boolean checkStorageGroupByPath(PartialPath path) {
   *
   * @param prefixPath a prefix path or a full path, may contain '*'.
   */
-  List<PartialPath> getAllTimeseriesPathWithAlias(PartialPath prefixPath) throws MetadataException {
+  Pair<List<PartialPath>, Integer> getAllTimeseriesPathWithAlias(PartialPath prefixPath, int limit, int offset) throws MetadataException {


Please update the Javadoc...

JackieTien97 · 2020-10-23T07:09:24Z

sonarcloud · 2020-10-23T07:20:06Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities (and 0 Security Hotspots to review)
1 Code Smell

No Coverage information
0.0% Duplication

The version of Java (1.8.0_252) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11.
Read more here

JackieTien97 added 3 commits June 15, 2020 17:45

init

9b904d0

OOM

c90ee47

cast long

b2e094f

qiaojialin reviewed Jun 17, 2020

View reviewed changes

merge with master

87e3e1c

jixuan1989 reviewed Jun 29, 2020

View reviewed changes

SilverNarcissus reviewed Jul 8, 2020

View reviewed changes

qiaojialin added the memory label Sep 2, 2020

JackieTien97 added 10 commits September 9, 2020 11:15

some changes

cbf3b83

resolve conflicts

aade12a

some changes

4181276

resolve conflicts

efcfbd3

solve function parameter

c1443ca

resolve conflicts

e329b25

remove useless limit in logical plan generating

084bf02

delete useless constructor in ShowTimeSeriesPlan

99c6fe8

add limit in logical select

31a0549

solve soffset

13bdd2f

neuyilan reviewed Oct 14, 2020

View reviewed changes

change back JDBCExample and add more tips in execption

fac0b6e

Alima777 reviewed Oct 16, 2020

View reviewed changes

JackieTien97 added 2 commits October 19, 2020 17:09

add hot change support

e954a6c

resolve conflicts

c03bfc6

Alima777 requested changes Oct 20, 2020

View reviewed changes

fix conflicts

e74f241

SteveYurongSu reviewed Oct 21, 2020

View reviewed changes

add javadoc

00e820a

resolve conflicts

897c663

HTHou merged commit 2600bf8 into master Oct 23, 2020

qiaojialin deleted the OOMImprove branch October 23, 2020 17:11

SteveYurongSu mentioned this pull request Oct 29, 2020

[IOTDB-749] Handle select * from root OOM #1884

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[IOTDB-749] Handle select * from root OOM #1367

[IOTDB-749] Handle select * from root OOM #1367

JackieTien97 commented Jun 15, 2020

qiaojialin Jun 17, 2020

sonarcloud bot commented Jun 29, 2020

jixuan1989 Jun 29, 2020

jixuan1989 Jun 29, 2020

jixuan1989 commented Jun 29, 2020

SilverNarcissus left a comment

SilverNarcissus Jul 8, 2020

neuyilan Oct 14, 2020

Alima777 Oct 20, 2020

Alima777 left a comment

Alima777 left a comment

Alima777 Oct 20, 2020

Alima777 Oct 20, 2020

JackieTien97 Oct 21, 2020

SteveYurongSu Oct 21, 2020

SteveYurongSu Oct 21, 2020

JackieTien97 commented Oct 23, 2020

sonarcloud bot commented Oct 23, 2020

[IOTDB-749] Handle select * from root OOM #1367

[IOTDB-749] Handle select * from root OOM #1367

Conversation

JackieTien97 commented Jun 15, 2020

Choose a reason for hiding this comment

sonarcloud bot commented Jun 29, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jixuan1989 commented Jun 29, 2020

SilverNarcissus left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Alima777 left a comment

Choose a reason for hiding this comment

Alima777 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JackieTien97 commented Oct 23, 2020

sonarcloud bot commented Oct 23, 2020