Skip to content
This repository has been archived by the owner on Jan 24, 2024. It is now read-only.

Add Batch support #42

Merged
merged 6 commits into from
Oct 4, 2019
Merged

Add Batch support #42

merged 6 commits into from
Oct 4, 2019

Conversation

jiazhai
Copy link
Contributor

@jiazhai jiazhai commented Sep 18, 2019

Fix #16
In KoP we would like to keep batch message. In Kafka message passed into Broker in format Records, each Records contains 1 or more Record. This is similar to the batched Message in Pulsar.
But because we have to turn Kafka Records into Pulsar BatchedMessage to make message could both be recognized by both Pulsar and Kafka client , we have to read each Record out from Records, and turn into Pulsar Message, This may involve some overhead of un-batch/re-batch.

changes:

  • move message produce/consume logic from KafkaRequestHandler.java into separate files.
  • add support for batch produce/consume.
  • add support for message headers.
  • change offset format in MessageRecordUtils.java to support batch index.
  • add test for added code.

@jiazhai jiazhai requested a review from sijie September 18, 2019 01:55
@jiazhai jiazhai self-assigned this Sep 18, 2019
@jiazhai jiazhai changed the title Add Batch support [WIP]Add Batch support Sep 18, 2019
@jiazhai jiazhai changed the title [WIP]Add Batch support Add Batch support Sep 19, 2019
@sijie sijie merged commit 4a88999 into streamnative:master Oct 4, 2019
@sijie sijie added this to the 0.0.1 milestone Oct 4, 2019
@sijie sijie deleted the batch_support branch October 4, 2019 20:22
@jiazhai jiazhai modified the milestones: 0.0.1, 0.1.0 Feb 17, 2020
michaeljmarshall pushed a commit to michaeljmarshall/kop that referenced this pull request Jan 24, 2023
Demogorgon314 pushed a commit to Demogorgon314/kop that referenced this pull request Nov 28, 2023
The wiremock dependency cannot be used with JDK 8, we need to use
wiremock-jre8 instead.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

handle batch messages when record is write to and read from pulsar
2 participants