You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
the idea: poll many topics at once with a pattern instead of one per call. something like *.*.*.*.<domain>.<tenant>, matching every topic that fits, including ones created later. NATS, MQTT and Solace all expose this (as subjects / topic filters) and people lean on it for event-driven systems.
we've had a couple of asks for it now, the latest wanting to swap a NATS event bus for iggy.
this isn't about storage. matching happens at poll time against live metadata, so there is nothing new to persist (and naming topics a.b.c.d already works today).
we could fake it client-side: list topics, filter, poll each. but then you have to re-list constantly to catch new topics, and it's a lot of round trips. this feels like something we should own server-side: a poll takes a pattern, the server resolves it against current metadata and returns messages from every matching topic.
the interesting questions:
offsets: the per-topic/partition cursors already exist, but a multi-topic poll response has to carry many of them at once (today's response is single-topic).
consumer groups: a newly matched topic has no committed offset for the group, so who gets its partitions, and where do they start reading? (the rebalance / revocation rabbit hole)
syntax: token wildcards like NATS/MQTT (*, >/#) vs full regex. i lean token, since it's predictable and cheap to match.
scope: do we match within a single stream, or across streams?
no rush on timing. realistically this lands after VSR, and i'd put it roughly a notch below #2854 in priority. mostly i'm opening this to see who wants it, what their topic patterns look like, and to settle the questions above before anyone writes code.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
-
the idea: poll many topics at once with a pattern instead of one per call. something like
*.*.*.*.<domain>.<tenant>, matching every topic that fits, including ones created later. NATS, MQTT and Solace all expose this (as subjects / topic filters) and people lean on it for event-driven systems.we've had a couple of asks for it now, the latest wanting to swap a NATS event bus for iggy.
this isn't about storage. matching happens at poll time against live metadata, so there is nothing new to persist (and naming topics
a.b.c.dalready works today).we could fake it client-side: list topics, filter, poll each. but then you have to re-list constantly to catch new topics, and it's a lot of round trips. this feels like something we should own server-side: a poll takes a pattern, the server resolves it against current metadata and returns messages from every matching topic.
the interesting questions:
*,>/#) vs full regex. i lean token, since it's predictable and cheap to match.no rush on timing. realistically this lands after VSR, and i'd put it roughly a notch below #2854 in priority. mostly i'm opening this to see who wants it, what their topic patterns look like, and to settle the questions above before anyone writes code.
(moved out of #3505, which i'm closing.)
thoughts?
Beta Was this translation helpful? Give feedback.
All reactions