KAFKA-3787: Preserve the message timestamp in mirror maker #1466
Conversation
|
@becketqin, any reason not to do this? cc @junrao Assuming that this is the right thing to do, we should include a test. |
|
Yes, that makes sense. We should include this in the 0.10.0 branch. |
|
@xiaotao183, are you willing to add a test to cover the fix? |
|
@ijuma Sure, will do |
|
@ijuma test case added |
| @@ -673,9 +673,9 @@ object MirrorMaker extends Logging with KafkaMetricsGroup { | |||
| def handle(record: BaseConsumerRecord): util.List[ProducerRecord[Array[Byte], Array[Byte]]] | |||
| } | |||
|
|
|||
| private object defaultMirrorMakerMessageHandler extends MirrorMakerMessageHandler { | |||
| object defaultMirrorMakerMessageHandler extends MirrorMakerMessageHandler { | |||
There was a problem hiding this comment.
You can leave it as private[tools] instead of public.
|
@ijuma please review again |
| assertEquals("topic", producerRecord.topic()) | ||
| assertNull(producerRecord.partition()) | ||
| assertEquals("key", new String(producerRecord.key())) | ||
| assertEquals("value", new String(producerRecord.value())) |
There was a problem hiding this comment.
Style nitpick: can you please remove the () from producerRecord.timestamp, producerRecord.topic, etc.? There are no side-effects, so there is no need for ().
|
@ijuma done |
|
@xiaotao183 Thanks! LGTM, merging to 0.10.0 and trunk. |
|
Sorry, I just realised that the test is in |
…ore/src/test/scala/unit/kafka/tools
|
@ijuma done |
The timestamp of messages consumed by mirror maker is not preserved after sending to target cluster. The correct behavior is to keep create timestamp the same in both source and target clusters. Author: Tao Xiao <xiaotao183@gmail.com> Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk> Closes #1466 from xiaotao183/KAFKA-3787 (cherry picked from commit f4a263b) Signed-off-by: Ismael Juma <ismael@juma.me.uk>
|
Thanks, tests passed locally. Merged to trunk and 0.10.0 branches. |
The timestamp of messages consumed by mirror maker is not preserved after sending to target cluster. The correct behavior is to keep create timestamp the same in both source and target clusters. Author: Tao Xiao <xiaotao183@gmail.com> Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk> Closes apache#1466 from xiaotao183/KAFKA-3787 (cherry picked from commit f4a263b) Signed-off-by: Ismael Juma <ismael@juma.me.uk>
The timestamp of messages consumed by mirror maker is not preserved after sending to target cluster. The correct behavior is to keep create timestamp the same in both source and target clusters.