Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kafka Test Subscriber doesn't match Consumer Group behaviour #1412

Closed
sinnwise opened this issue Apr 30, 2024 · 2 comments · Fixed by #1413
Closed

Kafka Test Subscriber doesn't match Consumer Group behaviour #1412

sinnwise opened this issue Apr 30, 2024 · 2 comments · Fixed by #1413
Labels
AioKafka Issues related to `faststream.kafka` module Confluent Issues related to `faststream.confluent` module Kafka Issues related to Kafka functionality (confluent and aiokafka both)

Comments

@sinnwise
Copy link

sinnwise commented Apr 30, 2024

Currently the Kafka Test Publisher / Subscriber combo returns early upon hitting a subscriber / topic combination.

If an application has two subscribers, across two consumer-groups listening on a single topic, FastStream does not respect that both subscribers should receive a message each from the topic. In testing only the first-matched topic would run.

for handler in self.broker._subscribers.values(): # pragma: no branch
if topic in handler.topics:
return await call_handler(
handler=handler,
message=[incoming]
if isinstance(handler, AsyncAPIBatchSubscriber)
else incoming,
rpc=rpc,
rpc_timeout=rpc_timeout,

Removing the return and letting each Subscriber handle the alert seems to get me closer to working behaviour (essentially we will have to create a list of subscribers attached to consumer groups and run handle on the first one in each consumer group – not sure how filters should come into play here).

Will work on a PR soon ☺️

@sifex
Copy link
Contributor

sifex commented May 1, 2024

PR opened on my non-work account.

@Lancetnik
Copy link
Collaborator

Thank you for the Issue and for the PR as well! I'll take a look close soon, sorry for waiting

@Lancetnik Lancetnik added Confluent Issues related to `faststream.confluent` module AioKafka Issues related to `faststream.kafka` module Kafka Issues related to Kafka functionality (confluent and aiokafka both) labels May 16, 2024
github-merge-queue bot pushed a commit that referenced this issue May 25, 2024
…ren't being respected (#1413)

* Commit for #1413

* Fixed bug in returned value

* Fixed bug in returned value

* Merged in from Main

* Merged in from Main

* Refectored Code slightly

* linting

* linting + formatting

* tests: remove useless markers

* Added batch tests for group_id overlap versions of subscribers

* Added batch tests for group_id overlap versions of subscribers

---------

Co-authored-by: Nikita Pastukhov <diementros@yandex.ru>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
AioKafka Issues related to `faststream.kafka` module Confluent Issues related to `faststream.confluent` module Kafka Issues related to Kafka functionality (confluent and aiokafka both)
Projects
None yet
3 participants