Add feature to automatically remove compaction configurations for inactive datasources #11232
Conversation
This is to be consistent with the other similar existing functionality (i.e. |
| * under the License. | ||
| */ | ||
|
|
||
| package org.apache.druid.java.util; |
There was a problem hiding this comment.
Why did you choose to put this class in this package?
I didn't find an equivalent class in the java.util package.
I think for now it might be better to move this to a private class in KillCompactionConfig which appears to be the only place where this is used
There was a problem hiding this comment.
I think it would be useful for people using the RetryUtils (which is also in org.apache.druid.java.util). RetryUtils.retry retry condition requires an exception to be thrown and applying a predicate to the thrown exception. For cases where the task method does not throw an exception, the method can throw this RetryableException so that the RetryUtils can then retry
There was a problem hiding this comment.
Ah, this is what I suspected, but the package naming threw me off. Can you please add javadocs explaining this use case
| import java.util.function.Function; | ||
| import java.util.stream.Collectors; | ||
|
|
||
| public class KillCompactionConfig implements CoordinatorDuty |
Add feature to automatically remove compaction configurations for inactive datasources
Description
We currently already have tasklog auto cleanup (#3677) and audit logs auto cleanup (#11084). This PR adds a similar auto cleanup but for the compaction configurations (which are stored in the druid config table) to auto clean up compaction configs belonging to datasource that is no longer active -- meaning that the datasource has no used/unused segments.
This is useful when Druid user has a high churn of task / datasource in a short amount of time causing the metadata store size to grow uncontrollably.
This PR has: