[BUG] After the hertzbeat application is restarted, a random collector cannot be registered.

[leim]

Is there an existing issue for this?

#1688

• I have searched the existing issues

Current Behavior

随机某一个或几个collector是离线状态，需要手动重启该collector，就会注册成功。

Expected Behavior

所有的collector都应该是在线状态，不需要手动重启

Steps To Reproduce

1、hertzbeat主服务正常运行，注册有20个collector
2、重启hertzbeat主服务
3、观察collector的状态

Environment

HertzBeat version(s):1.4.3和1.5.0都试验过，都会出现该问题

Debug logs

2024-04-03 11:09:58 [netty-client-worker-13] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1712113798398.
2024-04-03 11:10:03 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1712113803373.
2024-04-03 11:10:03 [netty-client-worker-13] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1712113803402.
2024-04-03 11:10:08 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1712113808373.
2024-04-03 11:10:08 [netty-client-worker-13] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1712113808398.
2024-04-03 11:10:13 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1712113813373.
2024-04-03 11:10:13 [netty-client-worker-13] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1712113813398.
2024-04-03 11:10:18 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1712113818373.
2024-04-03 11:10:18 [netty-client-worker-13] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1712113818398.
2024-04-03 11:10:23 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1712113823373.
2024-04-03 11:10:23 [netty-client-worker-13] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1712113823397.
2024-04-03 11:10:28 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1712113828373.
2024-04-03 11:10:28 [netty-client-worker-13] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1712113828398.
2024-04-03 11:10:33 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1712113833373.
2024-04-03 11:10:33 [netty-client-worker-13] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1712113833400.
2024-04-03 11:10:38 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1712113838373.
2024-04-03 11:10:38 [netty-client-worker-13] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1712113838397.
2024-04-03 11:10:43 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1712113843373.
2024-04-03 11:10:43 [netty-client-worker-13] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1712113843399.

Anything else?

No response

[tomsun28]

thanks for feedback.

[tomsun28]

repeat with issue #1688

[Ceilzcx]

@leim i think it is a bug, do you still have the log files from that day?
I'd like to see a more detailed log, now just some heartbeat message, had some warn or error msg?

[leim]

首先，master node 启动成功

2024-04-25 09:03:04 [main] INFO org.dromara.hertzbeat.manager.Manager - Started Manager in 35.88 seconds (JVM running for 37.609) 2024-04-25 09:03:04 [main] INFO org.dromara.hertzbeat.manager.service.impl.NoticeConfigServiceImpl - load default notice template in internal jar 2024-04-25 09:03:04 [main] INFO org.dromara.hertzbeat.manager.service.impl.AppServiceImpl - load define path /opt/hertzbeat/define 2024-04-25 09:03:06 [main] INFO org.dromara.hertzbeat.collector.collect.database.JdbcSpiLoader - start load jdbc drivers 2024-04-25 09:03:06 [main] INFO org.dromara.hertzbeat.collector.collect.database.JdbcSpiLoader - end load jdbc drivers

然后是一大堆这样的日志：

2024-04-25 09:03:11 [main] ERROR org.dromara.hertzbeat.manager.scheduler.CollectorAndJobScheduler - there is no collector name: xxxx online to assign job. 2024-04-25 09:03:11 [main] ERROR org.dromara.hertzbeat.manager.scheduler.CollectorAndJobScheduler - there is no collector name: xxxx online to assign job.

再过一段时间，会出现这样的日志，表明edge node 开始注册成功

2024-04-25 09:03:31 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1] REGISTERED 2024-04-25 09:03:31 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1] BIND: 0.0.0.0/0.0.0.0:1158 2024-04-25 09:03:31 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] ACTIVE 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0xaffe4081, L:/172.31.0.8:1158 - R:/42.159.124.31:3654] 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0x67c56ff0, L:/172.31.0.8:1158 - R:/42.159.124.31:3660] 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0x771f9138, L:/172.31.0.8:1158 - R:/42.159.124.31:3651] 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0xfad02302, L:/172.31.0.8:1158 - R:/42.159.124.31:3656] 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0x29c3756a, L:/172.31.0.8:1158 - R:/42.159.124.31:3658] 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0x1ef19861, L:/172.31.0.8:1158 - R:/42.159.124.31:3663] 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0x1dcbd910, L:/172.31.0.8:1158 - R:/42.159.124.31:3659] 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0x6ac4a123, L:/172.31.0.8:1158 - R:/42.159.124.31:3666] 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0xee872653, L:/172.31.0.8:1158 - R:/42.159.124.31:3670] 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ COMPLETE 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0xdce497e3, L:/172.31.0.8:1158 - R:/42.159.124.31:3665] 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ COMPLETE 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0x05292309, L:/172.31.0.8:1158 - R:/42.159.124.31:3662] 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ COMPLETE 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0xb3bf243d, L:/172.31.0.8:1158 - R:/42.159.124.31:3652] 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ COMPLETE 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0x639b86d0, L:/172.31.0.8:1158 - R:/42.159.124.31:3664]

然后提示这样：

2024-04-25 09:03:45 [netty-server-worker-3] INFO org.dromara.hertzbeat.manager.scheduler.netty.process.CollectorOnlineProcessor - the collector xxxx actively requests to go online. 2024-04-25 09:03:45 [netty-server-worker-2] INFO org.dromara.hertzbeat.manager.scheduler.netty.process.CollectorOnlineProcessor - the collector xxxx actively requests to go online.

此时再看“采集集群”，绝大多数节点已经注册成功并显示“在线”，会随机出现几个“离线”，

待系统稳定后，观察 master 节点，日志输出如下：

2024-04-25 09:08:29 [netty-server-worker-1] INFO org.dromara.hertzbeat.manager.scheduler.netty.process.HeartbeatProcessor - the collector xxxxx is not online. 2024-04-25 09:08:30 [warehouse-persistent-data-storage] INFO org.dromara.hertzbeat.warehouse.store.HistoryVictoriaMetricsDataStorage - [warehouse victoria-metrics] flush metrics data 335707961005824 is null, ignore. 2024-04-25 09:08:34 [netty-server-worker-1] INFO org.dromara.hertzbeat.manager.scheduler.netty.process.HeartbeatProcessor - the collector xxxxx is not online. 2024-04-25 09:08:39 [netty-server-worker-1] INFO org.dromara.hertzbeat.manager.scheduler.netty.process.HeartbeatProcessor - the collector xxxxx is not online. 2024-04-25 09:08:44 [netty-server-worker-1] INFO org.dromara.hertzbeat.manager.scheduler.netty.process.HeartbeatProcessor - the collector xxxxx is not online. 2024-04-25 09:08:49 [netty-server-worker-1] INFO org.dromara.hertzbeat.manager.scheduler.netty.process.HeartbeatProcessor - the collector xxxxx is not online. 2024-04-25 09:08:53 [warehouse-persistent-data-storage] INFO org.dromara.hertzbeat.warehouse.store.HistoryVictoriaMetricsDataStorage - [warehouse victoria-metrics] flush metrics data 345200261891840 is null, ignore. 2024-04-25 09:08:53 [warehouse-persistent-data-storage] INFO org.dromara.hertzbeat.warehouse.store.HistoryVictoriaMetricsDataStorage - [warehouse victoria-metrics] flush metrics data 339243678443264 is null, ignore. 2024-04-25 09:08:54 [netty-server-worker-1] INFO org.dromara.hertzbeat.manager.scheduler.netty.process.HeartbeatProcessor - the collector xxxxx is not online.

观察 edge 节点，日志如下

2024-04-25 09:08:24 [netty-client-worker-3] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1714007304531. 2024-04-25 09:08:29 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1714007309517. 2024-04-25 09:08:29 [netty-client-worker-3] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1714007309531. 2024-04-25 09:08:34 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1714007314517. 2024-04-25 09:08:34 [netty-client-worker-3] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1714007314530. 2024-04-25 09:08:39 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1714007319517. 2024-04-25 09:08:39 [netty-client-worker-3] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1714007319531. 2024-04-25 09:08:44 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1714007324517. 2024-04-25 09:08:44 [netty-client-worker-3] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1714007324531. 2024-04-25 09:08:49 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1714007329517. 2024-04-25 09:08:49 [netty-client-worker-3] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1714007329531. 2024-04-25 09:08:54 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1714007334517. 2024-04-25 09:08:54 [netty-client-worker-3] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1714007334534. 2024-04-25 09:08:59 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1714007339517.

此时重启edge 节点，观察master节点的日志：

2024-04-25 09:17:37 [warehouse-persistent-data-storage] INFO org.dromara.hertzbeat.warehouse.store.HistoryVictoriaMetricsDataStorage - [warehouse victoria-metrics] flush metrics data 335707961005824 is null, ignore. 2024-04-25 09:17:39 [netty-server-worker-1] INFO org.dromara.hertzbeat.manager.scheduler.netty.process.HeartbeatProcessor - the collector xxxxx is not online. 2024-04-25 09:17:59 [status-page-calculate-0] INFO org.dromara.hertzbeat.manager.component.status.CalculateStatus - start to calculate status page state 2024-04-25 09:18:00 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0x5735ca2a, L:/172.31.0.8:1158 - R:/42.159.124.31:3661] 2024-04-25 09:18:00 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ COMPLETE 2024-04-25 09:18:00 [netty-server-worker-2] INFO org.dromara.hertzbeat.manager.scheduler.netty.process.CollectorOnlineProcessor - the collector xxxxxxx actively requests to go online.

观察edge节点日志：

2024-04-25 09:17:58 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-1159"] 2024-04-25 09:17:58 [main] INFO org.springframework.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 1159 (http) with context path '' 2024-04-25 09:17:58 [main] INFO org.dromara.hertzbeat.collector.Collector - Started Collector in 6.828 seconds (JVM running for 12.334) 2024-04-25 09:17:58 [main] INFO org.dromara.hertzbeat.collector.collect.database.JdbcSpiLoader - start load jdbc drivers 2024-04-25 09:17:59 [main] INFO cc.blynk.clickhouse.ClickHouseDriver - Driver registered 2024-04-25 09:17:59 [main] INFO org.dromara.hertzbeat.collector.collect.database.JdbcSpiLoader - end load jdbc drivers 2024-04-25 09:18:05 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1714007885448. 2024-04-25 09:18:05 [netty-client-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1714007885462. 2024-04-25 09:18:10 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1714007890445. 2024-04-25 09:18:10 [netty-client-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1714007890459. 2024-04-25 09:18:15 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1714007895445. 2024-04-25 09:18:15 [netty-client-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1714007895460. 2024-04-25 09:18:20 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1714007900445. 2024-04-25 09:18:20 [netty-client-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1714007900460. 2024-04-25 09:18:25 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1714007905445.

[Ceilzcx]

首先，master node 启动成功

2024-04-25 09:03:04 [main] INFO org.dromara.hertzbeat.manager.Manager - Started Manager in 35.88 seconds (JVM running for 37.609) 2024-04-25 09:03:04 [main] INFO org.dromara.hertzbeat.manager.service.impl.NoticeConfigServiceImpl - load default notice template in internal jar 2024-04-25 09:03:04 [main] INFO org.dromara.hertzbeat.manager.service.impl.AppServiceImpl - load define path /opt/hertzbeat/define 2024-04-25 09:03:06 [main] INFO org.dromara.hertzbeat.collector.collect.database.JdbcSpiLoader - start load jdbc drivers 2024-04-25 09:03:06 [main] INFO org.dromara.hertzbeat.collector.collect.database.JdbcSpiLoader - end load jdbc drivers

然后是一大堆这样的日志：

2024-04-25 09:03:11 [main] ERROR org.dromara.hertzbeat.manager.scheduler.CollectorAndJobScheduler - there is no collector name: xxxx online to assign job. 2024-04-25 09:03:11 [main] ERROR org.dromara.hertzbeat.manager.scheduler.CollectorAndJobScheduler - there is no collector name: xxxx online to assign job.

再过一段时间，会出现这样的日志，表明edge node 开始注册成功

2024-04-25 09:03:31 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1] REGISTERED 2024-04-25 09:03:31 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1] BIND: 0.0.0.0/0.0.0.0:1158 2024-04-25 09:03:31 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] ACTIVE 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0xaffe4081, L:/172.31.0.8:1158 - R:/42.159.124.31:3654] 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0x67c56ff0, L:/172.31.0.8:1158 - R:/42.159.124.31:3660] 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0x771f9138, L:/172.31.0.8:1158 - R:/42.159.124.31:3651] 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0xfad02302, L:/172.31.0.8:1158 - R:/42.159.124.31:3656] 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0x29c3756a, L:/172.31.0.8:1158 - R:/42.159.124.31:3658] 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0x1ef19861, L:/172.31.0.8:1158 - R:/42.159.124.31:3663] 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0x1dcbd910, L:/172.31.0.8:1158 - R:/42.159.124.31:3659] 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0x6ac4a123, L:/172.31.0.8:1158 - R:/42.159.124.31:3666] 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0xee872653, L:/172.31.0.8:1158 - R:/42.159.124.31:3670] 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ COMPLETE 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0xdce497e3, L:/172.31.0.8:1158 - R:/42.159.124.31:3665] 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ COMPLETE 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0x05292309, L:/172.31.0.8:1158 - R:/42.159.124.31:3662] 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ COMPLETE 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0xb3bf243d, L:/172.31.0.8:1158 - R:/42.159.124.31:3652] 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ COMPLETE 2024-04-25 09:03:44 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0x639b86d0, L:/172.31.0.8:1158 - R:/42.159.124.31:3664]

然后提示这样：

2024-04-25 09:03:45 [netty-server-worker-3] INFO org.dromara.hertzbeat.manager.scheduler.netty.process.CollectorOnlineProcessor - the collector xxxx actively requests to go online. 2024-04-25 09:03:45 [netty-server-worker-2] INFO org.dromara.hertzbeat.manager.scheduler.netty.process.CollectorOnlineProcessor - the collector xxxx actively requests to go online.

此时再看“采集集群”，绝大多数节点已经注册成功并显示“在线”，会随机出现几个“离线”，

待系统稳定后，观察 master 节点，日志输出如下：

2024-04-25 09:08:29 [netty-server-worker-1] INFO org.dromara.hertzbeat.manager.scheduler.netty.process.HeartbeatProcessor - the collector xxxxx is not online. 2024-04-25 09:08:30 [warehouse-persistent-data-storage] INFO org.dromara.hertzbeat.warehouse.store.HistoryVictoriaMetricsDataStorage - [warehouse victoria-metrics] flush metrics data 335707961005824 is null, ignore. 2024-04-25 09:08:34 [netty-server-worker-1] INFO org.dromara.hertzbeat.manager.scheduler.netty.process.HeartbeatProcessor - the collector xxxxx is not online. 2024-04-25 09:08:39 [netty-server-worker-1] INFO org.dromara.hertzbeat.manager.scheduler.netty.process.HeartbeatProcessor - the collector xxxxx is not online. 2024-04-25 09:08:44 [netty-server-worker-1] INFO org.dromara.hertzbeat.manager.scheduler.netty.process.HeartbeatProcessor - the collector xxxxx is not online. 2024-04-25 09:08:49 [netty-server-worker-1] INFO org.dromara.hertzbeat.manager.scheduler.netty.process.HeartbeatProcessor - the collector xxxxx is not online. 2024-04-25 09:08:53 [warehouse-persistent-data-storage] INFO org.dromara.hertzbeat.warehouse.store.HistoryVictoriaMetricsDataStorage - [warehouse victoria-metrics] flush metrics data 345200261891840 is null, ignore. 2024-04-25 09:08:53 [warehouse-persistent-data-storage] INFO org.dromara.hertzbeat.warehouse.store.HistoryVictoriaMetricsDataStorage - [warehouse victoria-metrics] flush metrics data 339243678443264 is null, ignore. 2024-04-25 09:08:54 [netty-server-worker-1] INFO org.dromara.hertzbeat.manager.scheduler.netty.process.HeartbeatProcessor - the collector xxxxx is not online.

观察 edge 节点，日志如下

2024-04-25 09:08:24 [netty-client-worker-3] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1714007304531. 2024-04-25 09:08:29 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1714007309517. 2024-04-25 09:08:29 [netty-client-worker-3] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1714007309531. 2024-04-25 09:08:34 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1714007314517. 2024-04-25 09:08:34 [netty-client-worker-3] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1714007314530. 2024-04-25 09:08:39 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1714007319517. 2024-04-25 09:08:39 [netty-client-worker-3] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1714007319531. 2024-04-25 09:08:44 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1714007324517. 2024-04-25 09:08:44 [netty-client-worker-3] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1714007324531. 2024-04-25 09:08:49 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1714007329517. 2024-04-25 09:08:49 [netty-client-worker-3] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1714007329531. 2024-04-25 09:08:54 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1714007334517. 2024-04-25 09:08:54 [netty-client-worker-3] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1714007334534. 2024-04-25 09:08:59 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1714007339517.

此时重启edge 节点，观察master节点的日志：

2024-04-25 09:17:37 [warehouse-persistent-data-storage] INFO org.dromara.hertzbeat.warehouse.store.HistoryVictoriaMetricsDataStorage - [warehouse victoria-metrics] flush metrics data 335707961005824 is null, ignore. 2024-04-25 09:17:39 [netty-server-worker-1] INFO org.dromara.hertzbeat.manager.scheduler.netty.process.HeartbeatProcessor - the collector xxxxx is not online. 2024-04-25 09:17:59 [status-page-calculate-0] INFO org.dromara.hertzbeat.manager.component.status.CalculateStatus - start to calculate status page state 2024-04-25 09:18:00 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ: [id: 0x5735ca2a, L:/172.31.0.8:1158 - R:/42.159.124.31:3661] 2024-04-25 09:18:00 [netty-server-boss-0] INFO io.netty.handler.logging.LoggingHandler - [id: 0x8fd6dbd1, L:/0.0.0.0:1158] READ COMPLETE 2024-04-25 09:18:00 [netty-server-worker-2] INFO org.dromara.hertzbeat.manager.scheduler.netty.process.CollectorOnlineProcessor - the collector xxxxxxx actively requests to go online.

观察edge节点日志：

2024-04-25 09:17:58 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-1159"] 2024-04-25 09:17:58 [main] INFO org.springframework.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 1159 (http) with context path '' 2024-04-25 09:17:58 [main] INFO org.dromara.hertzbeat.collector.Collector - Started Collector in 6.828 seconds (JVM running for 12.334) 2024-04-25 09:17:58 [main] INFO org.dromara.hertzbeat.collector.collect.database.JdbcSpiLoader - start load jdbc drivers 2024-04-25 09:17:59 [main] INFO cc.blynk.clickhouse.ClickHouseDriver - Driver registered 2024-04-25 09:17:59 [main] INFO org.dromara.hertzbeat.collector.collect.database.JdbcSpiLoader - end load jdbc drivers 2024-04-25 09:18:05 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1714007885448. 2024-04-25 09:18:05 [netty-client-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1714007885462. 2024-04-25 09:18:10 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1714007890445. 2024-04-25 09:18:10 [netty-client-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1714007890459. 2024-04-25 09:18:15 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1714007895445. 2024-04-25 09:18:15 [netty-client-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1714007895460. 2024-04-25 09:18:20 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1714007900445. 2024-04-25 09:18:20 [netty-client-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.processor.HeartbeatProcessor - collector receive manager server response heartbeat, time: 1714007900460. 2024-04-25 09:18:25 [heartbeat-worker-0] INFO org.dromara.hertzbeat.collector.dispatch.entrance.CollectServer - collector send cluster server heartbeat, time: 1714007905445.

thank you for you reply. i will fix in 5.1