[IOTDB-3928][IOTDB-4097]New Compaction Performer —— Fast Compaction #7621
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ng sensor schema in fast cross compaction performer, which will cause adding resource read lock more then one time and not releasing it later
… the unseq data has been strengthened to improve the compaction efficiency.
…paction performer in inner space compaction
…kPerformerTests to use fast compaction performer instead
…eration, which can speed up compaction
…urrent chunk or page is not large enough
THUMarkLau
requested changes
Nov 22, 2022
| } | ||
|
|
||
| @Override | ||
| public void excute() |
There was a problem hiding this comment.
Suggested change
| public void excute() | |
| public void execute() |
| public void excute() | ||
| throws PageException, IllegalPathException, IOException, WriteProcessException { | ||
| compactFiles(); | ||
| if (hasStartMeasurement) { |
There was a problem hiding this comment.
As startNewtMeasurement is an individual method, there should be a method called endMeasurement too.
There was a problem hiding this comment.
rename to setNewMeasurement.
| public class FileElement { | ||
| public TsFileResource resource; | ||
|
|
||
| public boolean isOverlap = false; |
There was a problem hiding this comment.
Suggested change
| public boolean isOverlap = false; | |
| public boolean isSelected = false; |
| throws IOException, PageException, WriteProcessException, IllegalPathException { | ||
| while (!chunkMetadataQueue.isEmpty()) { | ||
| ChunkMetadataElement firstChunkMetadataElement = chunkMetadataQueue.peek(); | ||
| List<ChunkMetadataElement> overlappedChunkMetadatas = |
There was a problem hiding this comment.
Suggested change
| List<ChunkMetadataElement> overlappedChunkMetadatas = | |
| List<ChunkMetadataElement> overlappedChunkMetadataList = |
|
|
||
| public long priority; | ||
|
|
||
| public boolean isOverlaped = false; |
There was a problem hiding this comment.
Suggested change
| public boolean isOverlaped = false; | |
| public boolean isSelected= false; |
| * <p>Notice: if is aligned chunk, return true if any of value chunk has data been deleted. Return | ||
| * false if and only if all value chunks has no data been deleted. | ||
| */ | ||
| protected boolean isChunkModified(ChunkMetadataElement chunkMetadataElement) { |
There was a problem hiding this comment.
Change this function to a method of ChunkMetadataElement itself
Comment on lines
+311
to
+324
| while (pointPriorityReader.hasNext() | ||
| && pointPriorityReader.currentPoint().getTimestamp() < nextPageElement.startTime) { | ||
| // write data point to chunk writer | ||
| compactionWriter.write(pointPriorityReader.currentPoint(), subTaskId); | ||
| pointPriorityReader.next(); | ||
| if (candidateOverlappedPages.size() > oldSize) { | ||
| // during the process of writing overlapped points, if the first page is compacted | ||
| // completely or a new chunk is deserialized, there may be new pages overlapped with the | ||
| // first page in page queue which are added into the list. If so, the next overlapped | ||
| // page in the list may be changed, so we should re-get next overlap page here. | ||
| oldSize = candidateOverlappedPages.size(); | ||
| nextPageElement = candidateOverlappedPages.get(0); | ||
| } | ||
| } |
There was a problem hiding this comment.
The iterator pattern here is not written properly. The correct iterator pattern would call hasNext, then next, and then current, but here it's not that order. So it's easy to think that the last data point is going to be lost here.
THUMarkLau
approved these changes
Nov 25, 2022
…date device time in endChunkGroup method
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New Compaction Performer, which speeds up the cross space compaction, including aligned and nonAligned timeseries.