Issue 2638: Compaction Limits #2670
Conversation
| long end = start; | ||
| long timeDiff = 0; |
There was a problem hiding this comment.
Do we need the end and the timeDiff here? Seems we can only break the following loop if (currentTime-start) >= limit?
There was a problem hiding this comment.
The timediff is held for reporting after loop exit when debugging is enabled.
| } | ||
| if (meta.getUsage() >= threshold || (limit > 0 && timeDiff > limit) || !running) { | ||
| // We allow the usage limit calculation to continue so that we get a accurate | ||
| // report of where the usage was prior to running compaction. | ||
| continue; |
There was a problem hiding this comment.
We should break the loop if exceed the max compaction time? all the remaining logs will be skipped.
There was a problem hiding this comment.
The consumer, would not get any indication of how may logs were not compacted if we skip out of loop. This ensures that the exit report of the usage buckets give a complete picture of what was there and what was compacted. The usage gathering from what I saw as extremely fast compared to the compaction. The logs show that this does not take a appreciable amount of time. However, a consumer whom is tuning the system may need this insight.
klwilson227
force-pushed
the
2638-Compaction-Limits
branch
from
07ac04c
to
f9dd093
Compare
klwilson227
force-pushed
the
2638-Compaction-Limits
branch
from
f9dd093
to
4820aed
Compare
This resolves issue 2638, by allowing a customer to set a limit on the duration of the major and minor compaction runs. This allows the customer to balance the needs for compaction against the need for normal garbage collection. Unbounded execution of compaction can starve the garbage collection process, and may lead to extended periods of high disk IO. By setting a upper bound on the execution duration for the garbage collection, we can ensure a balance is maintained between both GC and Compaction processes.
Logging has been improved to allow a user to see the complete usage of the buckets, vs. those that were compacted, so that they can decide on any further tuning that may be required.
Test cases have been added to ensure limits are enforced properly and that defaulting of the values is correctly implemented.