[IOTDB-5]Combine deletion with FileNodeProcessor and FileNodeManager #7

jt2594838 · 2019-01-20T07:01:45Z

Implemented deletion-related interfaces in FileNodeProcessor and FileNodeManager.

# Conflicts: # iotdb/src/test/java/org/apache/iotdb/db/engine/bufferwrite/BufferWriteProcessorNewTest.java # iotdb/src/test/java/org/apache/iotdb/db/engine/bufferwrite/BufferWriteProcessorTest.java

# Conflicts: # iotdb/src/main/java/org/apache/iotdb/db/engine/filenode/FileNodeProcessor.java # iotdb/src/main/java/org/apache/iotdb/db/engine/overflow/ioV2/OverflowResource.java # iotdb/src/main/java/org/apache/iotdb/db/engine/overflow/ioV2/OverflowSupport.java

iotdb/src/main/java/org/apache/iotdb/db/engine/version/SimpleFileVersionController.java

iotdb/src/test/java/org/apache/iotdb/db/engine/modification/DeletionFileNodeTest.java

jixuan1989 · 2019-02-15T09:19:35Z

iotdb/src/main/java/org/apache/iotdb/db/engine/version/SimpleFileVersionController.java

+    File[] versionFiles = directory.listFiles((dir, name) -> name.startsWith(FILE_PREFIX));
+    File versionFile = null;
+    if (versionFiles != null && versionFiles.length > 0) {
+      Arrays.sort(versionFiles, Comparator.comparing(File::getName));


Suppose there are two version files: 900 and 1000. Then Comparator.comparing(File.getName) is incorrect.

Replaced with scan-and-find-maximum.

jixuan1989 · 2019-02-15T12:24:41Z

iotdb/src/main/java/org/apache/iotdb/db/engine/bufferwrite/BufferWriteProcessor.java

+    if (isFlush) {
+      // flushing MemTable cannot be directly modified since another thread is reading it
+      flushMemTable = flushMemTable.copy();
+      flushMemTable.delele(deviceId, measurementId, timestamp);


Hm... so is this operation is just for guaranteeing queryBufferWriteData() ?
Actually, flushMemTable = flushMemTable.copy(); can not affect the flushing memtable... so you have to add the Deletion operation in the modification file at the same time.

iotdb/src/main/java/org/apache/iotdb/db/engine/filenode/FileNodeProcessor.java

jixuan1989 · 2019-02-16T11:51:49Z

iotdb/src/main/java/org/apache/iotdb/db/engine/filenode/FileNodeProcessor.java

      statMonitor.registStatistics(statStorageDeltaName, this);
    }
+    try {
+      versionController = new SimpleFileVersionController(fileNodeDirPath);


How about create a VersionControllerFactory.
Latter we can modify the factory to switch the implementation of the versionController, otherwith SysTimestampVersionController is meaningless..

Not quite necessary, SysTimestampVersionController is for test.

iotdb/src/main/java/org/apache/iotdb/db/engine/memtable/AbstractMemTable.java

jixuan1989 · 2019-02-16T12:30:13Z

iotdb/src/main/java/org/apache/iotdb/db/engine/memtable/AbstractMemTable.java

+   * @return A reduced copy of chunk if chunk contains data with timestamp less than 'timestamp',
+   * of null.
+   */
+  private IWritableMemChunk filterChunk(IWritableMemChunk chunk, long timestamp) {


how about using the concept of Page?
Suppose a Chunk consists of page1, page2, page3,...., page n, and you will delete page 1, 2 and a part of page 3.
Then, just re-compress page 3 (and as a result, this page will have just a few points), and keep page 4~n into the new Chunk.

Good idea, but unfortunately IWritableMemChunk does not provide such interfaces. I would suggest you issue this and improve it later.

jixuan1989 · 2019-02-16T12:53:50Z

tsfile/src/main/java/org/apache/iotdb/tsfile/write/TsFileWriter.java

        }

-        fileWriter.endChunkGroup(chunkGroupFooter);
+        fileWriter.endChunkGroup(chunkGroupFooter, version++);


Suppose we have many TsFiles, and they are generated by different programs..
When we put all of them into an IoTDB instance (e.g., using Postback module), maybe we need a version repairing tool .
(There is no need to add the function into this PR, but if so, we need record an issue at least.)

It is not a big issue. Stand-alone TsFile does not support deletion. After importing all the TsFiles, any new deletion takes effect on all of them. So the only thing we need to do is to make the version of new deletions greater than those of all existing TsFiles, which is straight-forward.

# This is the 1st commit message: Fix query one sensor in a vector from memtable (apache#3056) Fix query one sensor in a vector from memtable # This is the commit message apache#2: [IOTDB-1310] Enable docker, docker-compose and testcontainer for End to end test (apache#3024) * enable TestCongtainer for E2E test for (singleNode and cluster) * remove duplicated operations in integration-test phase * move spotless:apply to a profile `spotless`, which is enabled by default. Co-authored-by: xiangdong huang <sainthxd@gmail.com> # This is the commit message apache#3: add sink interface # This is the commit message apache#4: 3 new event sinks # This is the commit message apache#5: add ts sink # This is the commit message apache#6: add ts sink # This is the commit message apache#7: add mqtt sink # This is the commit message apache#8: refactor sink module # This is the commit message apache#9: init package sink.manager # This is the commit message apache#10: rename SinkException # This is the commit message apache#11: remove id in config # This is the commit message apache#12: fix doc # This is the commit message apache#13: add test framework # This is the commit message apache#14: add alertmanager sink & test & doc # This is the commit message apache#15: beautify the doc

update

jt2594838 added 10 commits January 20, 2019 10:30

move from old repo

d072e9e

move tests from old repo

f1cabf8

move srcs from the old repo

4a38abe

move tests from the old repo

6559289

fix BufferWriteProcessor

ba0bed9

fix MemTableFlushUtils

9cd5d11

fix tsfile

1bf0ddf

fix tsfile writer

a6dbdbc

multiple fixes

8af13f1

revert the delete order: disk first then memory

fb5802d

jt2594838 changed the title ~~delete develop phase 2~~ delete development phase 2 Jan 20, 2019

jt2594838 added 2 commits January 20, 2019 15:14

fix the header error in package-info of modification

dbcf2ed

Merge branch 'delete_dev1' into delete_dev2

e6ead42

jt2594838 changed the title ~~delete development phase 2~~ Combine deletion with FileNodeProcessor and FileNodeManage Jan 20, 2019

jt2594838 changed the title ~~Combine deletion with FileNodeProcessor and FileNodeManage~~ Combine deletion with FileNodeProcessor and FileNodeManager Jan 21, 2019

jt2594838 changed the title ~~Combine deletion with FileNodeProcessor and FileNodeManager~~ [IOTDB-5]Combine deletion with FileNodeProcessor and FileNodeManager Jan 21, 2019

jt2594838 and others added 11 commits January 21, 2019 15:45

Merge branch 'delete_main' into delete_dev1

ebbf9f8

Merge branch 'delete_dev1' into delete_dev2

df9fd23

Merge branch 'delete_main' into delete_dev1

4740d95

Merge branch 'delete_dev1' into delete_dev2

a2f3ec8

# Conflicts: # iotdb/src/test/java/org/apache/iotdb/db/engine/bufferwrite/BufferWriteProcessorNewTest.java # iotdb/src/test/java/org/apache/iotdb/db/engine/bufferwrite/BufferWriteProcessorTest.java

Merge branch 'delete_main' into delete_dev1

0cb7a98

Merge branch 'delete_dev1' into delete_dev2

12adb38

Merge branch 'delete_main' into delete_dev1

2ffc23a

replace the file header with standard Apache Header

228f614

add hashCode() method

014aa86

Merge remote-tracking branch 'origin/delete_dev1' into delete_dev2

5c971b4

jixuan1989 reviewed Feb 15, 2019

View reviewed changes

iotdb/src/main/java/org/apache/iotdb/db/engine/version/SimpleFileVersionController.java Show resolved Hide resolved

jixuan1989 reviewed Feb 15, 2019

View reviewed changes

iotdb/src/test/java/org/apache/iotdb/db/engine/modification/DeletionFileNodeTest.java Outdated Show resolved Hide resolved

jixuan1989 reviewed Feb 15, 2019

View reviewed changes

fix sonar tips

ebcdccf

jixuan1989 reviewed Feb 15, 2019

View reviewed changes

jixuan1989 reviewed Feb 16, 2019

View reviewed changes

iotdb/src/main/java/org/apache/iotdb/db/engine/filenode/FileNodeProcessor.java Show resolved Hide resolved

jixuan1989 reviewed Feb 16, 2019

View reviewed changes

iotdb/src/main/java/org/apache/iotdb/db/engine/memtable/AbstractMemTable.java Outdated Show resolved Hide resolved

jixuan1989 reviewed Feb 16, 2019

View reviewed changes

iotdb/src/main/java/org/apache/iotdb/db/engine/memtable/AbstractMemTable.java Show resolved Hide resolved

jixuan1989 reviewed Feb 16, 2019

View reviewed changes

xiangdong huang and others added 2 commits February 16, 2019 20:58

fix Apache Header in some files

0986aff

fix by comments

ba51118

jixuan1989 closed this Feb 19, 2019

jt2594838 deleted the delete_dev2 branch February 21, 2019 03:36

SpriCoder referenced this pull request in SpriCoder/iotdb Jul 11, 2021

Merge pull request #7 from apache/master

886576b

update

xufwind95 mentioned this pull request Mar 14, 2022

数据库执行一段时间后，有大量假死的查询出现 #5215

Closed

[IOTDB-5]Combine deletion with FileNodeProcessor and FileNodeManager #7

[IOTDB-5]Combine deletion with FileNodeProcessor and FileNodeManager #7

Uh oh!

Conversation

jt2594838 commented Jan 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jt2594838 Feb 17, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jt2594838 commented Jan 20, 2019 •

edited

Loading

jt2594838 Feb 17, 2019 •

edited

Loading