balance leader command keep running forever

[kikimo]

Please check the FAQ documentation before raising an issue

Describe the bug (required)

balance leader command keep running forever;

Your Environments (required)

• OS: uname -a
• Compiler: g++ --version or clang++ --version
• CPU: lscpu
• Commit id ce15bca

How To Reproduce(required)

Steps to reproduce the behavior:

1. create a cluster with 3storage
2. create a space with 512 parts + 3 replicas
3. add two new storeage and run balance leader

we observed that the balance leader command run for more than one days and its still running as showed below:

(root@nebula) [ttos_3p3r]> show jobs;
+--------+------------------+-----------+----------------------------+----------------------------+
| Job Id | Command          | Status    | Start Time                 | Stop Time                  |
+--------+------------------+-----------+----------------------------+----------------------------+
| 5      | "ZONE_BALANCE"   | "QUEUE"   |                            |                            |
| 4      | "DATA_BALANCE"   | "QUEUE"   |                            |                            |
| 3      | "LEADER_BALANCE" | "RUNNING" | 2022-01-04T08:16:03.000000 |                            |
| 2      | "LEADER_BALANCE" | "FAILED"  | 2022-01-04T08:05:19.000000 | 2022-01-04T08:05:24.000000 |
+--------+------------------+-----------+----------------------------+----------------------------+
Got 4 rows (time spent 1677/2182 us)

Wed, 05 Jan 2022 15:01:53 CST

Expected behavior

Additional context

[critical27]

Probably same problem with #3742

[Sophie-Xie]

fix in #3668