Implmeneted data split for kafka message

[jeongkyun-oh]

Proposed changes

• Data split is implemented to publish Kafka message
• Key is not implemented for deciding which partition the message is sent to
• Header includes:
  • checksum for the data consistency
  • the number of total segments for a message
  • segment index

Types of changes

Please put an x in the boxes related to your change.

• Bugfix
• New feature or enhancement
• Others

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

• I have read the CONTRIBUTING GUIDELINES doc
• I have signed the CLA
• Lint and unit tests pass locally with my changes ($ make test)
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)
• Any dependent changes have been merged and published in downstream modules

[yoomee1313]

Could you please add some comments in TestKafka_split and TestKafka_makeProducerMessage function? It would be easier to understand the tests with more comments!

[jeongkyun-oh]

Could you please add some comments in TestKafka_split and TestKafka_makeProducerMessage function? It would be easier to understand the tests with more comments!

Alright. I will add some comments. e9e1d2a

[joowon-byun]

I have some questions and comments.

1. Why splitting is required in Kafka? What are the benefits?
2. I think publishing segments in parallel can enhance speed.
3. A data is sent by calling publish() one time, and is received by calling several subscribe(). How do we know if the received data is a splitted one? After receiving, the segments need to be combined into one.
   It would be nice to specify a protocol including data ID and segment number. Also, one publish() call should be correspond to one subscribe().

[jeongkyun-oh]

I have some questions and comments.

1. Why splitting is required in Kafka? What are the benefits?
2. I think publishing segments in parallel can enhance speed.
3. A data is sent by calling publish() one time, and is received by calling several subscribe(). How do we know if the received data is a splitted one? After receiving, the segments need to be combined into one.
   It would be nice to specify a protocol including data ID and segment number. Also, one publish() call should be correspond to one subscribe().

@winnie-byun Thanks for your comments.

1. I've got the feedback that the large message would affect Kafka's performance badly. The benchmark said that the throughput would be worse if the larger messages are published logarithmically.
2. The order would be important, so I think parallelism is not a good option.
3. Yes, you're right. This PR includes only the producer part. The consumer part will be updated soon. The header includes all the information for resembling data (segment index, total segments, checksum). Do you think this is not enough? I will commit the data ID part as well in this PR. Also, Publish corresponds to Consume. Subscribe keeps consuming in a loop. What do you think?

[joowon-byun]

Thanks for the kind explaination. I didn't see makeProducerMessage before.
This code looks good to be :)

[KimKyungup]

LGTM except minor comments