-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
help request: Pubsub with Kafka ready for production ? #10947
Comments
Push will not occur (due to the complexity of Nginx workers and some long connections). The existing mode is req/resp, where the client tells the gateway from which offset position to start sending subsequent messages. The input parameters of fetch message is topic/partition/offset, which will send the latest message starting from offset (if there are too many messages and exceed the maximum byte limit, the number of messages that meet the response body size will be returned). After the client receives the response, it can traverse it to retrieve all messages, and in the last message, you will get the timestamp and offset of this message. The client can use them to retrieve newer messages in the next fetch request. |
Thanks for your answer, |
Hi @arnauddeman, I have checked with a few of the maintainers. Pub/Sub is still experimental, requiring the client to use WebSocket rather than HTTP or Kafka Protocol. I have also checked the issues you raised at https://github.com/apache/apisix/issues/created_by/arnauddeman, but I would like to understand your requirements and scenarios. Can you share the details? (what kind of problems do you want to resolve? e.g., proxy Kafka behind APISIX? or configuring OIDC with Kafka?). This can help us better understand your needs and may provide a suggestion. |
Hi @juzhiyuan, As you suggested, I have tested the use of a web socket route, connected to a locally developed API which uses a Kafka client. The associated APISIX route uses the OIDC plugin and the local API can retrieve the token to determine the user and filter the Kafka messages. |
Hi @arnauddeman, I have checked with @bzp2010 and @moonming; we need some time to discuss this. Will keep you updated. |
Description
Hello, I would like to know the status of PubSub with Kafka implementation in Apisix. Is it an experimental feature or is it ready for production ? The underlying Kafka Client lua-resty-kafka is still flagged as experimental.
My second question is about the implementation of the pubsub pattern in Apisix : the new messages are not pushed to the client only the messages published before the PubSubQuery are. Is this the expected behaviour ?
Environment
APISIX docker version, image apache/apisix:3.6.0-debian
The text was updated successfully, but these errors were encountered: