[Improve][DataProxy] Add DataProxy node load information

[gosonzhang]

Description

When the system is running, the SDK will pull the list of DataProxy node addresses of the cluster from the Manager, and select several DataProxy from the list for data reporting processing. With the operation of the system, DataProxy's load will be changed due to changes in the number of connected SDK connections and the amount of messages sent by the business. It is possible that the load of DataProxy will become worse and cannot provide external services, or the load becomes lower which no SDK connecting, and it's service capability cannot be effectively exerted.

In order to solve this problem, I want to add the load information to the DataProxy node, and expose this load information to the Manager and SDK:

1. DataProxy regularly counts its own load, and reports the load information to the Manager through heartbeat;
2. The Manager updates stores the load information reported by each DataProxy, and provides it to the SDK when the SDK requests the DataProxy information;
3. Based on the DataProxy IP list and load information provided by the Manager, the SDK randomly selects the remaining DataProxy nodes after excluding the Top N DataProxy nodes with excessive load, and performs data transmission services; the SDK periodically obtains the DataProxy ip and load information from the Manager;
4. After the SDK establishes a connection with DataProxy, it decides whether to switch DataProxy for data reporting through the load information returned by DataProxy in the heartbeat response;
5. The SDK operates according to 3, when selecting a new DataProxy to establish a connection to report data, avoid the DataProxy node of Top N with excessive load.

The whole idea is similar to the following picture:

Let's see if it is OK, if there is no problem, I will try to complete the processing of this part.

InLong Component

InLong Manager, InLong DataProxy, InLong SDK

Are you willing to submit PR?

• Yes, I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

This issue is stale because it has been open for 60 days with no activity.

[gosonzhang]

The issue is finished and closed