Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Looking to increase kafa support Routine Load kafka_topic_list support #30110

Open
2 of 3 tasks
jiugem opened this issue Jan 18, 2024 · 0 comments
Open
2 of 3 tasks
Labels
kind/feature Categorizes issue or PR as related to a new feature.

Comments

@jiugem
Copy link
Contributor

jiugem commented Jan 18, 2024

Search before asking

  • I had searched in the issues and found no similar issues.

Description

Routine Load 导入Kafka中的数据功能,现在是只支持1个topic。
对于分库分表场景,dts订阅会有许多topic,创建许多个Routine Load可能导致写入频次过高影响效率,配置也比较麻烦。
参考 ClickHouse 的Kafka表引擎,它是支持配置多个topic的,并行导入速度是非常快的。

Use case

Kafka引擎表配置
kafka_topic_list = 'APP_LOG',
kafka_group_name = 'toClickhouse',
kafka_format = 'JSONAsString',
kafka_num_consumers = 16,
kafka_thread_per_consumer = 1;

Related issues

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@jiugem jiugem added the kind/feature Categorizes issue or PR as related to a new feature. label Jan 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Categorizes issue or PR as related to a new feature.
Projects
None yet
Development

No branches or pull requests

1 participant