Skip to content
This repository has been archived by the owner on Jan 24, 2024. It is now read-only.

[FEATURE] Proxy support for KoP #57

Open
jiazhai opened this issue Dec 27, 2019 · 7 comments
Open

[FEATURE] Proxy support for KoP #57

jiazhai opened this issue Dec 27, 2019 · 7 comments
Assignees
@jiazhai
Labels
help wanted Extra attention is needed type/feature Indicates new functionality workflow::backlog

Comments

@jiazhai
Copy link
Contributor

jiazhai commented Dec 27, 2019 •
edited

Proxy support for KoP.
@jiazhai jiazhai added the type/feature Indicates new functionality label Dec 27, 2019
@jiazhai jiazhai self-assigned this Dec 31, 2019
@PierreZ
Copy link
Contributor

PierreZ commented Jan 9, 2020

Hi @jiazhai!

How did you plan to develop this feature? How can I help? 😄

@jiazhai jiazhai mentioned this issue Feb 14, 2020
Open
@jiazhai jiazhai added the help wanted Extra attention is needed label Jul 5, 2020
@jiazhai
Copy link
Contributor Author

jiazhai commented Jul 5, 2020

Currently this is still in our backlog. and not start working on it in our side. If anyone is interested to contribute to this PR welcome to comments on this issue. And Pierre already did some investigation on this, Thanks for @PierreZ's help on this.

@eolivelli
Copy link
Contributor

@jiazhai @PierreZ
is the Envoy proxy able to act like the Pulsar proxy on k8s ?

I mean, in Pulsar we have the "Pulsar proxy" that hides the complexity of the cluster inside k8s.

From the documentation I see that I should setup one Envoy proxy per each Pulsar broker, this is very awkward if you have a large Pulsar cluster, especially in case that the cluster is dynamically scaling up and down.

https://github.com/streamnative/kop/blob/master/docs/envoy-proxy.md

"For each broker with KoP enabled, a dependent Envoy proxy is required. Assuming that you have N brokers whose internal hostname is pulsar-broker-, where i is the broker ID that varies from 0 to N-1."

@BewareMyPower
Copy link
Collaborator

@eolivelli Yes, not like Pulsar proxy, Envoy acts more like a sidecar. Currently we have no proxy for KoP like Pulsar proxy.

@eolivelli
Copy link
Contributor

@BewareMyPower @merlimat @jiazhai @PierreZ
I have starting working on a Proxy for KOP.
I have also sent a PIP to dev@pulsar.apache.org in order to add protocol handlers in the Pulsar Proxy.

I will be happy to share my prototype and share my work upstream to this repository.

@BewareMyPower
Copy link
Collaborator

@eolivelli It would be great to have protocol handlers support in the Pulsar Proxy.

@eolivelli eolivelli mentioned this issue Sep 6, 2021
Merged
BewareMyPower pushed a commit that referenced this issue Sep 7, 2021
@eolivelli
Refactor APIs to allow access from the Proxy module (#710)
851a2cd 
This patch introduces a little refactor in order to support the implementation of a separate KOP Proxy Module:
- allow to access internal ByteBuf of KafkaHeaderAndRequest, in order to allow zero-copy request magement in the Proxy
- extract some static methods in order to use the same algorithm for mapping coordinators

Relates to #57 

Co-authored-by: Enrico Olivelli <eolivelli@apache.org>
@eolivelli
Copy link
Contributor

eolivelli commented Sep 8, 2021 •
edited

@jiazhai @BewareMyPower @PierreZ @sijie @merlimat
I have created this design document for the proxy that I am implementing.
#717

I have already implemented a working prototype, the performances with the proxy are the same as while using regular Pulsar proxy.
I have also already implemented support for Token Auth.

I will be happy to contribute the prototype to this repository and continue the development directly here and not in my repo.
This way it will be easier to maintain it and also to have more contributions.

BewareMyPower pushed a commit that referenced this issue Sep 8, 2021
@eolivelli @BewareMyPower
Refactor APIs to allow access from the Proxy module (#710)
14e2f1a 
This patch introduces a little refactor in order to support the implementation of a separate KOP Proxy Module:
- allow to access internal ByteBuf of KafkaHeaderAndRequest, in order to allow zero-copy request magement in the Proxy
- extract some static methods in order to use the same algorithm for mapping coordinators

Relates to #57 

Co-authored-by: Enrico Olivelli <eolivelli@apache.org>
michaeljmarshall added a commit to michaeljmarshall/kop that referenced this issue Apr 24, 2023
@michaeljmarshall
[security] Proxy to broker and broker to broker hostname verification (…
10f28df 
…streamnative#57)
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@jiazhai jiazhai

Labels
help wanted Extra attention is needed type/feature Indicates new functionality workflow::backlog
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@sijie @PierreZ @jiazhai @eolivelli @BewareMyPower