Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Namesrv nearby route #4382

Closed
lwclover opened this issue May 26, 2022 · 6 comments
Closed

Namesrv nearby route #4382

lwclover opened this issue May 26, 2022 · 6 comments

Comments

@lwclover
Copy link
Contributor

lwclover commented May 26, 2022

RocketMQ同机房就近生产和消费机制。

RocketMQ注册、发现机制

  • Broker
    • 间隔30秒发送broker上的Topic全量信息到所有Nameserv。broker启动的时候也是延迟注册到Namesrv上的。当有topic发生变更时TopicConfigManager.dataVersion发生变更。
    • 当有topic创建和更新时,broker立刻发送Topic增量信息到所有Namesrv。同时触发Namesrv的BrokerLiveInfo.dataVersion发生变更。
  • Namesrv
    • 保存集群和broker关系、broker信息(brokerName和地址)、broker真实连接和topic版本号,topic信息。
    • 调度线程每10秒钟检查一次,如果发现一个broker 120秒都没有更新注册信息,则删除和这个broker相关的所有信息。
  • Producer&Consumer
    • Producer第一次发送Message时,会同步向namesrv请求topic路由信息
    • Consumer启动过程中会向namesrv请求topic路由信息
    • 每隔30秒向namesrv拉取一次topic路由信息,根据路由信息内容(不是版本号)判断是否变化,发生变化更新
  • 总结:
    Producer第一次发送消息时会同步向namesrv请求topic路由信息,然后默认轮询往每个Queue发送消息。
    Consumer启动时获取topic路由信息,然后请求broker获取consumer实例列表,最后根据consumer的数量和负载均衡算法给consumer分配queue。

实现rocketmq同机房就近生产和消费:

1.broker增加一个zoneName标识配置,通过heartbeat注册到namesrv上。
2.Producer和Consumer可以通过API或者环境变量、系统变量设置zoneName和zoneMode(这是一个开关)。
3.Producer和Consumer获取topic路由信息时,在namesrv上返回对应标识的broker信息。

image

image

@dongeforever
Copy link
Member

Good Issue. Do you have any practice with this?

@lwclover
Copy link
Contributor Author

lwclover commented May 27, 2022

Good Issue. Do you have any practice with this?

In our company, Cross-VDC hypermetro needs to be implemented within a region,avoid faults in a single region.

An Apache RocketMQ Cluster is deployed across VDCS within a region. In normal cases, the nearest VDC produces and consumes messages. When a VDC is unavailable, the other VDC can produce and consume messages.

@dongeforever
Copy link
Member

@lwclover Is the master-slave replication cross-VDC too?

@lwclover
Copy link
Contributor Author

lwclover commented Jun 6, 2022

One case is across VDC on different clouds, and another case is across geographical locations within a VDC, like beijing and baoding

dongeforever pushed a commit that referenced this issue Aug 11, 2022
* fix:when broker is down,async send model can not retry

* fix Issue #3556

* fix Issue #3556

* fix Issue #3556

* fix Issue #3556

* async send model success retry when occurs Exception

* test case testSendMessageAsync_WithException

* modify test case testSendMessageAsync_WithException

* adding exception message with broker addr when occuring broker connect
timeout

* repair code style

* in async mode,set ThreadExecutorPool AbortPolicy

* Optimize asynchronous send timeout logic code

* Adjusting log information

* Delete SendMessageProcessor.java

* 恢复上一个版本

* namesrv nearby route

* Revert to previous version code

* add lisence header to Network.java

* code style

* 根据broker和client配置的标识返回路由信息

* Delete greetings.yml

* Delete .gitignore

* reset previous version

* reset to previous version

* code style

* Namesrv nearby route implements all by RPCHook

* reset to prevous version

* ServiceProvider return to original position

* Nearby route make the code clear

* revert to ori version

* reset to ori version

* reset to ori version

* reset to ori version

* Keep same as the official version

* Keep same as the official version

* Keep same as the official version

Co-authored-by: sunhangda <sunhangda@lixiang.com>
@github-actions
Copy link

github-actions bot commented Jul 9, 2023

This issue is stale because it has been open for 365 days with no activity. It will be closed in 3 days if no further activity occurs.

@github-actions github-actions bot added the stale label Jul 9, 2023
@github-actions
Copy link

This issue was closed because it has been inactive for 3 days since being marked as stale.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants