Agent cannot reconnect OAP

[liyanyan-dev]

Please answer these questions before submitting your issue.

• Why do you submit this issue?
• [√ ] Question or discussion
• Bug
• Requirement
• Feature or performance improvement

---

Question

I occasionally find that after the OAP is restarted, the agent can't be reconnected to the OAP for about half an hour. After reading the source code of the agent, I find that the following problems may exist:

1. Multiple threads will report errors and update the connection status. When a certain thread reports an error, it happens that the GRPCChannelManager thread starts scheduling，After changing the connection status to connected, another thread reports an error and changes the status to disconnected. In extreme cases, the agent cannot be reconnected with OAP all the time.

---

Bug

• Which version of SkyWalking, OS and JRE?
  skywalking 8.1.0
  centos7

• Which company or project?

• What happened?
  If possible, provide a way to reproduce the error. e.g. demo application, component version.

---

Requirement or improvement

• Please describe your requirements or improvement suggestions.

[kezhenxu94]

Multiple threads will report errors and update the connection status. When a certain thread reports an error, it happens that the GRPCChannelManager thread starts scheduling，After changing the connection status to connected, another thread reports an error and changes the status to disconnected. In extreme cases, the agent cannot be reconnected with OAP all the time.

How could that be possible if your OAP server is back to healthy, if you OAP server is healthy finally, all the threads should be able to finally change the status to connected. Otherwise there must be something else wrong. Do you have any extra information/logs?

[liyanyan-dev]

OAP server is healthy, other agent can reconnnect to oap;Only one agent can't reconnect，but restarting the agent is normal .The agent debug log always reports that it is unable to connect and the segment data is ignored;The log file was removed when the agent was restarted

[kezhenxu94]

Only one agent can't reconnect，but restarting the agent is normal.

What's the difference between that agent and the others? Is that agent reproducible always? If yes, can you minimize the codes for us to reproduce this issue and fix?

[wu-sheng]

If it is not connected, it should continue to retry. In this case, can't happen the race condition you described.