[Bug] [work-server] Client session timed out, have not heard from server

[Bingz2]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

[WARN] 2022-12-28 21:55:58.593 org.apache.zookeeper.ClientCnxn:[1108] - Client session timed out, have not heard from server in 20005ms for sessionid 0xff84f7a7634676d3
[INFO] 2022-12-28 21:55:58.594 org.apache.zookeeper.ClientCnxn:[1156] - Client session timed out, have not heard from server in 20005ms for sessionid 0xff84f7a7634676d3, closing socket connection and attempting reconnect
[INFO] 2022-12-28 21:55:58.709 org.apache.curator.framework.state.ConnectionStateManager:[251] - State change: SUSPENDED
[WARN] 2022-12-28 21:55:58.714 org.apache.dolphinscheduler.plugin.registry.zookeeper.ZookeeperConnectionStateListener:[50] - Registry suspended
[WARN] 2022-12-28 21:55:58.726 org.apache.dolphinscheduler.server.worker.registry.WorkerRegistryClient:[145] - registry connection state is SUSPENDED, ready to retry connection
[WARN] 2022-12-28 21:56:28.709 org.apache.curator.framework.state.ConnectionStateManager:[308] - Session timeout has elapsed while SUSPENDED. Injecting a session expiration. Elapsed ms: 30000. Adjusted session timeout ms: 30000
[WARN] 2022-12-28 21:56:28.714 org.apache.curator.ConnectionState:[376] - Session expired event received
[INFO] 2022-12-28 21:56:49.247 org.apache.zookeeper.ZooKeeper:[693] - Session: 0xff84f7a7634676d3 closed[INFO] 2022-12-28 21:56:49.247 org.apache.zookeeper.ZooKeeper:[442] - Initiating client connection, connectString=slave1.realtime.we:2181,slave2.realtime.we:2181,slave3.realtime.we:2181,slave4.realtime.we:2181,slave5.realtime.we:2181 sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@1eb9a3ef
[INFO] 2022-12-28 21:56:49.248 org.apache.curator.framework.state.ConnectionStateManager:[251] - State change: LOST
[INFO] 2022-12-28 21:56:49.248 org.apache.zookeeper.ClientCnxn:[522] - EventThread shut down for session: 0xff84f7a7634676d3
[WARN] 2022-12-28 21:56:49.248 org.apache.dolphinscheduler.plugin.registry.zookeeper.ZookeeperConnectionStateListener:[42] - Registry disconnected
[WARN] 2022-12-28 21:56:49.249 org.apache.dolphinscheduler.server.worker.registry.WorkerRegistryClient:[157] - registry connection state is DISCONNECTED, ready to stop myself
[INFO] 2022-12-28 21:56:49.250 org.apache.dolphinscheduler.server.worker.WorkerServer:[193] - worker server is stopping ..., cause : registry connection state is DISCONNECTED, stop myself
[INFO] 2022-12-28 21:56:49.251 org.apache.zookeeper.ClientCnxn:[1025] - Opening socket connection to server slave1.realtime.we/10.10.10.17:2181. Will not attempt to authenticate using SASL (unknown error)
[INFO] 2022-12-28 21:56:49.252 org.apache.zookeeper.ClientCnxn:[879] - Socket connection established to slave1.realtime.we/10.10.10.17:2181, initiating session
[INFO] 2022-12-28 21:56:49.253 org.apache.zookeeper.ClientCnxn:[1299] - Session establishment complete on server slave1.realtime.we/10.10.10.17:2181, sessionid = 0xff84f7a74886bdf6, negotiated timeout = 30000
[INFO] 2022-12-28 21:56:49.253 org.apache.curator.framework.state.ConnectionStateManager:[251] - State change: RECONNECTED
[INFO] 2022-12-28 21:56:52.289 org.apache.dolphinscheduler.remote.NettyRemotingServer:[243] - netty server closed
[INFO] 2022-12-28 21:56:52.290 org.apache.dolphinscheduler.server.worker.registry.WorkerRegistryClient:[173] - worker node : 10.10.10.50:1234 unRegistry from ZK /nodes/worker/default/10.10.10.50:1234.
[INFO] 2022-12-28 21:56:52.299 org.apache.dolphinscheduler.server.worker.registry.WorkerRegistryClient:[180] - heartbeat executor shutdown
[INFO] 2022-12-28 21:56:52.413 org.apache.curator.framework.imps.CuratorFrameworkImpl:[955] - backgroundOperationsLoop exiting
[INFO] 2022-12-28 21:56:52.414 org.apache.zookeeper.ClientCnxn:[522] - EventThread shut down for session: 0xff84f7a74886bdf6
[INFO] 2022-12-28 21:56:52.414 org.apache.zookeeper.ZooKeeper:[693] - Session: 0xff84f7a74886bdf6 closed
[INFO] 2022-12-28 21:56:52.415 org.apache.dolphinscheduler.server.worker.registry.WorkerRegistryClient:[183] - registry client closed
[INFO] 2022-12-28 21:56:52.477 org.apache.dolphinscheduler.remote.NettyRemotingClient:[390] - netty client closed
[INFO] 2022-12-28 21:56:52.478 org.apache.dolphinscheduler.service.alert.AlertClientService:[76] - alter client closed
[INFO] 2022-12-28 21:56:52.478 org.apache.dolphinscheduler.server.worker.WorkerServer:[239] - ready to kill all cache job, job size:0
[INFO] 2022-12-28 21:56:52.487 org.quartz.core.QuartzScheduler:[585] - Scheduler quartzScheduler_$NON_CLUSTERED paused.
[INFO] 2022-12-28 21:56:52.496 org.apache.dolphinscheduler.remote.NettyRemotingClient:[390] - netty client closed
[INFO] 2022-12-28 21:56:52.496 org.apache.dolphinscheduler.service.log.LogClientService:[74] - logger client closed
[INFO] 2022-12-28 21:56:52.497 org.springframework.scheduling.quartz.SchedulerFactoryBean:[845] - Shutting down Quartz Scheduler
[INFO] 2022-12-28 21:56:52.497 org.quartz.core.QuartzScheduler:[666] - Scheduler quartzScheduler$NON_CLUSTERED shutting down.
[INFO] 2022-12-28 21:56:52.497 org.quartz.core.QuartzScheduler:[585] - Scheduler quartzScheduler$NON_CLUSTERED paused.
[INFO] 2022-12-28 21:56:52.501 org.quartz.core.QuartzScheduler:[740] - Scheduler quartzScheduler$_NON_CLUSTERED shutdown complete.
[INFO] 2022-12-28 21:56:52.505 com.zaxxer.hikari.HikariDataSource:[350] - DolphinScheduler - Shutdown initiated...
[INFO] 2022-12-28 21:56:52.513 com.zaxxer.hikari.HikariDataSource:[352] - DolphinScheduler - Shutdown completed.
[INFO] 2022-12-28 21:56:52.514 org.apache.dolphinscheduler.server.worker.WorkerServer:[215] - springApplicationContext close

What you expected to happen

The work cannot connect to zk, but other work can connect to ZK

How to reproduce

work-server takes place almost every Wednesday night around 10 p.m

Anything else

No response

Version

2.0.x

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Thank you for your feedback, we have received your issue, Please wait patiently for a reply.

• In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
• If you haven't received a reply for a long time, you can join our slack and send your question to channel #troubleshooting

[brave-lee]

@Bingz2 Is it related to the load of the worker node？If the CPU of the worker node reaches 100% for a long time in a certain period, it will cause the heartbeat timeout with zk. I think this issue should be caused by this reason

[brave-lee]

I close this issue. If someone has other progress, we can open it or submit a new issue.