Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

如何让数据库和topic一一对应 #1716

Closed
ZSH9053 opened this issue Apr 17, 2019 · 4 comments
Closed

如何让数据库和topic一一对应 #1716

ZSH9053 opened this issue Apr 17, 2019 · 4 comments

Comments

@ZSH9053
Copy link

ZSH9053 commented Apr 17, 2019

我想让每个数据库都投递到和他名字一样的topic里,该如何设置,wiki我看了,不是很理解,
也做过很多尝试性的调整,都没有用,能否告知如何设置,谢谢
比如我数据库名字是A,B,C,topic名字也创建了A,B,C,如何配置

mq config

canal.mq.topic=A

dynamic topic route by schema or table regex

canal.mq.dynamicTopic=db_baseinfo_test
#canal.mq.partition=0

hash partition config

canal.mq.partitionsNum=8
canal.mq.partitionHash=.\..:id

@chenxyz707
Copy link
Contributor

在instance.properties里面配置
同步库配置
canal.instance.filter.regex=A\\..*,B\\..*,C\\..*

消息队列配置
使用正则路由schema和table
格式topic:schema.table,topic:schema.table,topic:schema.table
canal.mq.dynamicTopic=topic_A:A,topic_B:B,topic_C:C
最好配置一个canal.mq.topic作为默认的topic,在实践中发现,有一些mysql schema的binlog也会读进来(建表语句,grant语句等),如果没有这个默认的topic,会报找不到分区的错误,从而导致canal停止写入。
canal.mq.topic=a_default_topic

分区设置

  1. 如果每个topic只有一个分区设置如下
    canal.mq.partition=0
  2. 如果每个topic有多个分区,且希望按照表名做hash进行如下配置
    # hash partition config
    canal.mq.partitionsNum=3
    canal.mq.partitionHash=.*\\..*

@ZSH9053
Copy link
Author

ZSH9053 commented Apr 18, 2019

是不是canal.mq.topic这个参数随便写一个存在的topic,然后主要设置在这里对吧?canal.mq.dynamicTopic=topic_A:A,topic_B:B,topic_C:C

@chenxyz707
Copy link
Contributor

是的。canal.mq.topic算是一个默认的,没有匹配到的schema和table的日志都会输入到那个默认的里面去。消费的时候只关心你设置的动态分区。

@rewerma
Copy link
Collaborator

rewerma commented Apr 19, 2019

楼上正解

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants