[IOTDB-205]Support storage-group-level data ttl #429

jt2594838 · 2019-09-26T09:17:32Z

https://issues.apache.org/jira/browse/IOTDB-205

# Conflicts: # server/src/main/java/org/apache/iotdb/db/engine/merge/manage/MergeResource.java # server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java # server/src/main/java/org/apache/iotdb/db/service/TSServiceImpl.java

add show ttl

# Conflicts: # server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java # server/src/main/java/org/apache/iotdb/db/metadata/MTree.java # server/src/main/java/org/apache/iotdb/db/service/TSServiceImpl.java # server/src/test/java/org/apache/iotdb/db/engine/cache/DeviceMetaDataCacheTest.java # server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java # server/src/test/java/org/apache/iotdb/db/query/reader/ReaderTestHelper.java # server/src/test/java/org/apache/iotdb/db/query/reader/fileRelated/UnSealedTsFileReaderTest.java # server/src/test/java/org/apache/iotdb/db/query/reader/resourceRelated/SeqResourceReaderTest.java # server/src/test/java/org/apache/iotdb/db/query/reader/resourceRelated/UnseqResourceReaderTest.java # service-rpc/src/main/java/org/apache/iotdb/rpc/TSStatusCode.java

fix tests

change log level

jt2594838 · 2019-09-27T02:43:27Z

A design document in CHINESE.
数据定期删除设计文档.docx

jt2594838 · 2019-09-27T02:45:15Z

Experiment settings and results. Pay extra attention to the changes in file sizes.
config.properties.txt
iotdb-engine.properties.txt
insertTestWithDefaultPath_IoTDB_TTL_Write11569292133990.xlsx
SERVER_MODE_fit_10_2019_09_24.xlsx

# Conflicts: # jdbc/src/main/java/org/apache/iotdb/jdbc/Constant.java # jdbc/src/main/java/org/apache/iotdb/jdbc/IoTDBDatabaseMetadata.java # jdbc/src/main/java/org/apache/iotdb/jdbc/IoTDBMetadataResultSet.java # jdbc/src/main/java/org/apache/iotdb/jdbc/IoTDBStatement.java # server/src/main/java/org/apache/iotdb/db/engine/StorageEngine.java # server/src/main/java/org/apache/iotdb/db/metadata/MGraph.java # server/src/main/java/org/apache/iotdb/db/metadata/MManager.java # server/src/main/java/org/apache/iotdb/db/metadata/MTree.java # server/src/main/java/org/apache/iotdb/db/metadata/MetadataOperationType.java # server/src/main/java/org/apache/iotdb/db/qp/logical/Operator.java # server/src/main/java/org/apache/iotdb/db/service/TSServiceImpl.java # server/src/test/java/org/apache/iotdb/db/metadata/MManagerBasicTest.java # service-rpc/src/main/thrift/rpc.thrift

update comments fix license code refinements

LeiRui · 2019-10-15T13:13:30Z

Hi,
This is a review on functionality.
I find the operation unset TTL to <storage group> REALLY requires a seasoned user who knows exactly what he/she is doing. And I totally agree with what you have noted in the guidance:

Notice: When you set TTL to some storage groups, data out of the TTL will be made invisible
immediately, but because the data files may contain both out-dated and living data or the data files may be being used by queries, the physical removal of data is stale. If you increase or unset TTL
just after setting it previously, some previously invisible data may be seen again, but the
physically removed one is lost forever. In other word, different from delete statement, the
atomicity of data deletion is not guaranteed for efficiency concerns. So we recommend that you do not change the TTL once it is set or at least do not reset it frequently, unless you are determined to suffer the unpredictability.

I want to draw attention to another point that can be easily misunderstood when using the operation unset TTL to <storage group>. That is, the latest insertion time of a storage group, which is the split point used to determine whether the coming data point is sequential or unsequential, will not be “recycled" even if the data before now()-TTL are really deleted.

jt2594838 · 2019-10-16T08:31:05Z

That is true. We currently do not provide any way to "recycle" the last flushed time of a device other than DELETE STORAGE GROUP statements, even general DELETE DATA statements can not do that. And the "recycling" does not seem to bring many benefits since, on most occasions, data is rolling forward.

Actually, the data TTL has little relation to the last flushed time. And whether the data goes into sequence files or unsequence files should be invisible to users. I do not think users should be concerned about that.

What I have hidden in my implementation is that "UNSET TTL TO root.group1" equals "SET TTL TO root.group1 Long.maxValue". In all, data TTL just avoids the DBA deleting old data manually.

LeiRui · 2019-10-16T12:47:12Z

And whether the data goes into sequence files or unsequence files should be invisible to users.

I agree.

# Conflicts: # server/src/main/java/org/apache/iotdb/db/qp/executor/AbstractQueryProcessExecutor.java # server/src/main/java/org/apache/iotdb/db/qp/strategy/LogicalGenerator.java # server/src/main/java/org/apache/iotdb/db/qp/strategy/PhysicalGenerator.java # server/src/main/java/org/apache/iotdb/db/service/TSServiceImpl.java

server/src/main/java/org/apache/iotdb/db/engine/merge/manage/MergeResource.java

server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileProcessor.java

server/src/main/java/org/apache/iotdb/db/metadata/MetadataOperationType.java

server/src/main/java/org/apache/iotdb/db/qp/logical/Operator.java

server/src/main/java/org/apache/iotdb/db/utils/TestOnly.java

JackieTien97

Lambda used in several places should be replaced by inner class that implements the corresponding functional interface to avoid creating lambda class frequently.

jt2594838 · 2019-10-22T02:04:39Z

Creating a new inner class object or creating a lambda does not seem quite different to me. But I hope using method reference could help a bit.

# Conflicts: # server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java # server/src/main/java/org/apache/iotdb/db/service/TSServiceImpl.java

# Conflicts: # server/src/main/antlr3/org/apache/iotdb/db/sql/parse/TSLexer.g # server/src/main/antlr3/org/apache/iotdb/db/sql/parse/TSParser.g # server/src/main/java/org/apache/iotdb/db/qp/constant/SQLConstant.java # server/src/main/java/org/apache/iotdb/db/qp/strategy/LogicalGenerator.java

server/src/main/java/org/apache/iotdb/db/engine/merge/manage/MergeResource.java

fanhualta · 2019-10-23T08:41:50Z

Good job, please fix the conflicts.

docs/Documentation/UserGuide/5-IoTDB SQL Documentation/1-IoTDB Query Statement.md

server/src/main/java/org/apache/iotdb/db/engine/StorageEngine.java

server/src/main/java/org/apache/iotdb/db/qp/executor/AbstractQueryProcessExecutor.java

server/src/main/java/org/apache/iotdb/db/service/TSServiceImpl.java

# Conflicts: # docs/Documentation-CHN/UserGuide/4-Operation Manual/7-IoTDB Query Language.md # docs/Documentation/UserGuide/4-Operation Manual/7-IoTDB Query Statement.md # server/src/main/java/org/apache/iotdb/db/engine/StorageEngine.java # server/src/main/java/org/apache/iotdb/db/metadata/MGraph.java # server/src/main/java/org/apache/iotdb/db/metadata/MManager.java # server/src/main/java/org/apache/iotdb/db/metadata/MTree.java # server/src/main/java/org/apache/iotdb/db/monitor/StatMonitor.java # server/src/main/java/org/apache/iotdb/db/qp/executor/QueryProcessExecutor.java # server/src/main/java/org/apache/iotdb/db/service/TSServiceImpl.java # server/src/test/java/org/apache/iotdb/db/engine/modification/DeletionFileNodeTest.java # server/src/test/java/org/apache/iotdb/db/engine/modification/DeletionQueryTest.java # server/src/test/java/org/apache/iotdb/db/metadata/MManagerAdvancedTest.java # server/src/test/java/org/apache/iotdb/db/metadata/MManagerBasicTest.java # server/src/test/java/org/apache/iotdb/db/metadata/MManagerImproveTest.java # server/src/test/java/org/apache/iotdb/db/metadata/MTreeTest.java

# Conflicts: # server/src/main/java/org/apache/iotdb/db/conf/IoTDBConstant.java

# Conflicts: # jdbc/src/main/java/org/apache/iotdb/jdbc/Constant.java # jdbc/src/main/java/org/apache/iotdb/jdbc/IoTDBStatement.java # server/src/main/antlr3/org/apache/iotdb/db/sql/parse/TqlParser.g # server/src/main/java/org/apache/iotdb/db/metadata/MManager.java # server/src/main/java/org/apache/iotdb/db/metadata/MTree.java # server/src/main/java/org/apache/iotdb/db/qp/constant/SQLConstant.java # server/src/main/java/org/apache/iotdb/db/qp/executor/QueryProcessExecutor.java # server/src/main/java/org/apache/iotdb/db/qp/strategy/LogicalGenerator.java # server/src/main/java/org/apache/iotdb/db/qp/strategy/PhysicalGenerator.java # server/src/main/java/org/apache/iotdb/db/service/TSServiceImpl.java

docs/Documentation-CHN/UserGuide/4-Operation Manual/7-IoTDB Query Language.md

server/src/assembly/resources/conf/iotdb-engine.properties

server/src/main/java/org/apache/iotdb/db/engine/StorageEngine.java

server/src/main/java/org/apache/iotdb/db/engine/merge/manage/MergeResource.java

server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java

server/src/main/java/org/apache/iotdb/db/metadata/MGraph.java

qiaojialin · 2019-10-28T10:15:30Z

server/src/main/java/org/apache/iotdb/db/metadata/MGraph.java

-  static String combineMetadataInStrings(String[] metadatas) {
-    return MTree.combineMetadataInStrings(metadatas);
+  static String combineMetadataInStrings(String[] metadataArray) {
+    return MTree.combineMetadataInStrings(metadataArray);


is this method a devil?

What do you mean?

这个方法是魔鬼吗，这方法干嘛的，连个例子都没有

你问我我也不知道，我只是当了一下语法警察。

server/src/main/java/org/apache/iotdb/db/query/context/QueryContext.java

# Conflicts: # server/src/main/java/org/apache/iotdb/db/conf/IoTDBConstant.java # server/src/main/java/org/apache/iotdb/db/engine/StorageEngine.java # server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java # server/src/main/java/org/apache/iotdb/db/sync/receiver/SyncServiceImpl.java

jt2594838 added 20 commits September 12, 2019 17:22

add basic supports for TTL

80e917a

update antlr files

c456dad

add sql interface for ttl

73603eb

add timed TTL check task

afbd649

add tests

75e4eb5

remove unnecessary print

87d9946

add docs

db6268a

fix testTTLRead()

cccb8f0

fix ttl grammar and docs

f10899d

fix losing ttl after restart

499de7b

add show ttl

fix IoTDBTTLTest

233590d

fix tests

f66ca85

clean IoTDBStatement

b9782b6

add default ttl

485b144

change log level

38835ba

fix tests

add a test and logs

5a6a434

fix default ttl not effective

c5f2dcb

change log level

fix files in memory are not correctly removed and add tests

71b2278

jt2594838 added 4 commits September 27, 2019 13:34

fix word choice in the document.

84929dd

remove unnecessary dependency

3671695

update comments fix license code refinements

fix tests

f8c37f2