KAFKA-1981 Make log compaction point configurable

[ewasserman]

@jkreps

Implements control over the portion of the head of the log that will not be compacted (i.e. preserved in detail).

The log cleaner can be configured retain a minimum amount of the uncompacted "head" of the log.
This is enabled by setting one or more of the compaction lags:

log.cleaner.min.compaction.lag.ms
log.cleaner.min.compaction.lag.bytes
log.cleaner.min.compaction.lag.messages

with similar per topic configurations:

min.compaction.lag.ms
min.compaction.lag.bytes
min.compaction.lag.messages

These can be used to set constraints on the minimum message age, aggregate size, and/or count respectively that may be compacted. If none are set, all log segments are eligible for compaction except for the last segment, i.e. the one currently being written to. The active segment will not be compacted even if all of the compaction lag constraints are satisfied.

In particular this allows for the example use case described in the JIRA: "any consumer that is no more than 1 hour behind will get every message."

This contribution is my (Eric Wasserman's) original work and I license the work to the Kafka project under its open source license.

[ewasserman]

Test unrelated to this PR failed. Closing and reopening.

[ewasserman]

Tests passed locally. Re-opening.

[ewasserman]

closing request pending changes based on KIP-58 review.

[gwenshap]

@ewasserman hey, the vote passed a while back, but I'm not sure where things went from there... was there another PR that got merged?

[gwenshap]

never mind, found the second PR.

[ewasserman]

Gwen, the second PR is stalled waiting for KIP-33 (KAFKA-3163) to be merged.

On Aug 16, 2016, at 9:34 AM, Gwen (Chen) Shapira notifications@github.com wrote:

never mind, found the second PR.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub #1168 (comment), or mute the thread https://github.com/notifications/unsubscribe-auth/AFDbJNGS2rTCBeKcxED8SGYd-_hbuNbAks5qgeatgaJpZM4H8Tt3.