Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support multi-topic subscriptions with Apache Pulsar #8806

Open
laszlzso opened this issue May 31, 2022 · 3 comments
Open

Support multi-topic subscriptions with Apache Pulsar #8806

laszlzso opened this issue May 31, 2022 · 3 comments

Comments

@laszlzso
Copy link

There are multiple ways how this could be implemented, namely:

  1. Instead of/Beyond the existing configuration option "stream.pulsar.topic.name": "pulsar-test" we could have something like "stream.pulsar.topic.names": [ "pulsar-test-1", "pulsar-test-2"].
  2. Beyond the above existing configuration option we could have something like "stream.pulsar.topic.pattern": "pulsar-test-*" where the pattern adheres to standard regex rules.
  3. We could support such pattern-like inputs in the existing configuration option, e.g. an input field like "stream.pulsar.topic.name": "persistent://some_tenant/some_namespace/*" should make Pinot bind a real-time table to all topics under the given tenant/namespace.

Another alternative solution would be the capability of binding multiple topics to the same real-time table by sending subsequent requests to the Pinot Server (e.g., first, creating some_table as a real-time one and providing topic A as its source, however, a subsequent real-time table request with the name some_table and topic B would just subscribe to B and direct the corresponding flow into some_table, as well).

@Jackie-Jiang
Copy link
Contributor

cc @KKcorps

@estebanz01
Copy link

hola! 👋

any progress on this? I'm in a position where I'm creating multiple pulsar topics with the same schema and we need to have it in one table instead of one per topic.

@hpvd
Copy link

hpvd commented Apr 26, 2024

added this to [parent issue] Apache Pulsar related open topics #13010

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants