[IOTDB-3189] Fix compaction is not well-distributed across sgs #6324
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See IOTDB-3189.
In present IoTDB compaction task submitted order set by the
ICompactionTaskComparator
, currently this interface only hasDefaultCompactionTaskComparator
an implementation.DefaultCompactionTaskComparator
compares the compaction task priority with following several aspects:Different storage groups are not considered in this comparison. Therefore, when the tasks of different storage groups are compared side by side, it is easy to create uneven storage combinations. For example, if two SG groups commit a large number of compacton tasks in the sequence space with the same number of files, the same file size, and the same file level, the system may compact tasks in only one storage group due to the different submission timing.
To balance compaction tasks between different storage groups, when all other things being equal, each compaction task can be assigned a unique serial id that is counted individually by each virtual storage group and monotonically increases from zero. If two compaction tasks are from different storage groups, their sequence numbers are compared; the compaction task with a smaller sequence number has a higher priority. After adjustment, compaction priority comparison becomes