Implement Kafka protocol

[yanivbh1]

Summary

Implementation of Kafka protocol in Memphis

Value

To enable Kafka users to easily work with Memphis without changing their code, Memphis will support Kafka protocol

Persona(s)

• Data Engineers
• Developers
• Kafka users
• New Users

[g41797]

Hi
As far as I understand Q4 is target
I'd like to check possibility to participate

[yanivbh1]

Hey @g41797 ,
Thank you for your interest in contributing to Memphis!
As this feature is highly critical and should be greatly planned before, it would be great to also have more direct communication through our discord
Please let me know once your there, I will create a shared channel for us, and we will discuss the initial planning alongside @idanasulinmemphis

[g41797]

on board

[g41797]

Implementation of Kafka protocol in Memphis - client point of view

• Network protocol TCP
• Encryption - No
• Subset of APIs
• Kafka Client SDKs/libraries (installation,usage, configuration,tutorials)
• Memphis-Kafka documentation

Last is most important. It will help us to understand expectations of users and as result what we actually should to implement. I don't know exactly, but if Kafka and Memphis use different terminology (possibly for the same things) - explanation is the part of this doc. How will co-exist kafka-memphis and memhpis on the client side. Do we have some recommendations?

Simultaneously we need to look for ready solutions
e.g we know that go code will parse binary data with Kafka messages
how to simplify this process?
Redpanda uses JSON schema files for generation of c code

franz-go uses proprietary format for generation of go code.

May be mix of these solutions will be good idea?

[yanivbh1]

Thanks, @g41797 !
Before making that decision, it would be great to do some "mapping" work and
a) create a list of the Kafka API functions that would be relevant to Memphis (quorum-related functions, for example is not one of them)
b) create a mapping between Kafka functions to the equivalent Memphis function, and for those who left behind - create a placeholder/default behavior
c) Understand what would be the right pattern to implement a new protocol. Is it per API directly within the core or some sort of "adapter" that will act as the middleman before the broker and the clients, implementing different "drivers"

What do you think?

[g41797]

of course , I talked about code-generation just as part of brainstorming

as far as I understand dashboard is bestselling point of Memphis
how Kafka producers/consumers shown ? are there any difference? is it fully transparent? messages from Kafka producer may be consumed be memphis consumer etc
need smaltalk whatsapp/phone doesn't matter

regarding implementation

we are not talking about adapter on the client side, because in this case we need to support go, js, pyton etc a lot of works

"directly within the core" - best solution for performance but not in the early stage (stability etc)
separated adapter process looks better on this stage

as example of similar solution see nats-kafka

[idanasulin2706]

@g41797 client-side solution is indeed the worst option.
An adapter might cause a very poor performance what do you think will be the impact? also in terms of ordering and reliability (like ack etc.)

[g41797]

old joke: two programmers arguing whose program is better
first said - my program is faster
second - but my program works

kafka itself has misc adapters for "non-smart" clients
adapter is kind of sandbox, failure of adapter is not critical for core because separation. with right design developed code may be later used as plug-in within core

but why we are talking about design? i didn't see any reqs yet
usually when customer ask to support some feature he/she rare knows what he wants

please take a look on

• Features
• Overview

of course this information for developers, but it may be translated also to "customers language". we definitely need similar doc before deep dive

[idanasulin2706]

Totally agree, let's start from that part I'd say, after approving it we will be able to move forward. What do you think?

[g41797]

as old-fashioned man I think we need to talk

…

On Thu, Jun 1, 2023, 11:15 Idan Asulin ***@***.***> wrote: Totally agree, let's start from that part I'd say, after approving it we will be able to move forward. What do you think? — Reply to this email directly, view it on GitHub <#467 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACFRU6M4XOPXPXYXZHO2DTTXJBFSNANCNFSM6AAAAAAR2TA6JE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[yanivbh1]

Feature is being managed in issue #849