Implement KafkaSource controller

[slinkydeveloper]

Problem
We need to implement the KafkaSource controller for the data plane in this repo

Persona:
Which persona is this feature for? Users

Exit Criteria
A working kafka source implementation using the data-plane in this project. All e2e tests of kafka source passing.

Time Estimate (optional):
10

[matzew]

Interesting, could be aligned w/ the eventing-kafka repo? or is that competing ?

[slinkydeveloper]

The idea is to reuse the data plane we have here and build the control plane for it. I guess it's mostly competing, because by the nature of this data plane, this implementaiton will be a "multi-tenant" kafka source

[lionelvillard]

did you say multi-tenant, as of 1 data-plane component for all source instances? @slinkydeveloper ?

[slinkydeveloper]

@lionelvillard yep that's the idea. Of course this component can scale too

[lionelvillard]

Of course :-) Do you have a design document describing how the scaling works? I'm assuming it's using Keda, but I'm most curious to know how source instances scale independently to each other. @slinkydeveloper

[pierDipi]

Roughly the idea is to:

• add the number of consumers to run for each KafkaSource or Trigger (Egress in our terminology)
  • how?
    • KafkaSource: scale subresource (your PR) or an annotation
    • Trigger: an internal CRD with the scale subresource or an annotation
• use the RangeAssignor to partition consumer instances (your design document) + defining a total order for Egress configs so that each pod has the same view of the system.

[lionelvillard]

ah cool thanks!

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[pierDipi]

/remove-lifecycle stale