Skip to content

Add a new iterator policy for coordinator auto CompactionTask#10863

Closed
liuxiaohui1221 wants to merge 2 commits intoapache:masterfrom
liuxiaohui1221:feature-new-auto-compactionTask-policy
Closed

Add a new iterator policy for coordinator auto CompactionTask#10863
liuxiaohui1221 wants to merge 2 commits intoapache:masterfrom
liuxiaohui1221:feature-new-auto-compactionTask-policy

Conversation

@liuxiaohui1221
Copy link
Contributor

@liuxiaohui1221 liuxiaohui1221 commented Feb 8, 2021

Fixes #XXXX.

Description

This PR mainly fixes scenes where delayed data reaches a relatively serious level, and the efficiency of auto-compact task is low. The details are as follows:

  1. If a segment is smaller than inputSegmentSizeBytes, the segment size will be compact.In fact, we are more concerned with compact with multiple small files.
  2. The coordinator timer resubmits compact from the latest interval every time. There may be a situation that the old interval cannot do compact all the time due to insufficient slots.
  3. The Coordinator's timer submits in order from the latest interval to the oldest. Intervals submitted by the Coordinator may be blocked by a lock, and the Compact task that only applies the latest interval is more likely to block or fail and occupy a slot, we need to set skipOffsetFromLatest big enough, and it will waste recent many small files without compaction.

Main change list:

1.add new policy for auto compaction.
score
2.compactionTask not waiting.
3.compaction taskId changed
4.filter locked interval before submitted.
5.add some configuration parameters.

Renamed the class ...

Added a forbidden-apis entry ...

Added dynamic configuration parameters:

dynamic configuration detail default value
enableFilterLockedInterval whether or not filter locked interval before submitted,false/true false
minInputSegmentNum Compaction min input segments num 1
minorCompactRecentDays minor compaction are first search recent days segments to compact. P5D

Added coordination configuration parameters:

coordinator configuration detail default value
druid.coordinator.period.autoCompactionPeriod coordinator submit compactionTask period,it must smaller or equal than druid.coordinator.period.indexingPeriod PT30M
druid.coordinator.autoCompactionPolicy coordinator auto compaction iterator policy,newest or score newest

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

Key changed/added classes in this PR
  • MyFoo
  • OurBar
  • TheirBaz

2.compactionTask not waiting
3.taskId change
4.filter locked interval before submitted
5.add some parameters
@liuxiaohui1221 liuxiaohui1221 marked this pull request as draft February 18, 2021 06:20
@stale
Copy link

stale bot commented Apr 29, 2022

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If you think that's incorrect or this pull request should instead be reviewed, please simply write any comment. Even if closed, you can still revive the PR at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

@stale stale bot added the stale label Apr 29, 2022
@github-actions
Copy link

github-actions bot commented Oct 7, 2023

This pull request/issue has been closed due to lack of activity. If you think that
is incorrect, or the pull request requires review, you can revive the PR at any time.

@github-actions github-actions bot closed this Oct 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant