storage: BALANCE DATA failed with Transfer leader failed, status Unknown code -46

[veezhang]

BALANCE DATA 1620708035

 BALANCE DATA 1620708035
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+
| balanceId, spaceId:partId, src->dst                                                                                                                                                                         | status   |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+
| "[1620708035, 1:1, test1-1-3-storaged-1.test1-1-3-storaged-headless.nebulacluster-5088.svc.cluster.local:9779->test1-1-3-storaged-3.test1-1-3-storaged-headless.nebulacluster-5088.svc.cluster.local:9779]" | "FAILED" |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+
| "[1620708035, 1:2, test1-1-3-storaged-2.test1-1-3-storaged-headless.nebulacluster-5088.svc.cluster.local:9779->test1-1-3-storaged-3.test1-1-3-storaged-headless.nebulacluster-5088.svc.cluster.local:9779]" | "FAILED" |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+
| "[1620708035, 1:3, test1-1-3-storaged-0.test1-1-3-storaged-headless.nebulacluster-5088.svc.cluster.local:9779->test1-1-3-storaged-3.test1-1-3-storaged-headless.nebulacluster-5088.svc.cluster.local:9779]" | "FAILED" |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+
| "[1620708035, 1:5, test1-1-3-storaged-2.test1-1-3-storaged-headless.nebulacluster-5088.svc.cluster.local:9779->test1-1-3-storaged-3.test1-1-3-storaged-headless.nebulacluster-5088.svc.cluster.local:9779]" | "FAILED" |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+
| "Total:4, Succeeded:0, Failed:4, In Progress:0, Invalid:0"                                                                                                                                                  | 0.0      |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+

meta log

E0511 05:48:19.404897   128 BalanceTask.cpp:72] [1620708035, 1:5, test1-1-3-storaged-2.test1-1-3-storaged-headless.nebulacluster-5088.svc.cluster.local:9779->test1-1-3-storaged-3.test1-1-3-storaged-headless.nebulacluster-5088.svc.cluster.local:9779] Transfer leader failed, status Unknown code -46
E0511 05:48:19.405431   121 BalanceTask.cpp:72] [1620708035, 1:2, test1-1-3-storaged-2.test1-1-3-storaged-headless.nebulacluster-5088.svc.cluster.local:9779->test1-1-3-storaged-3.test1-1-3-storaged-headless.nebulacluster-5088.svc.cluster.local:9779] Transfer leader failed, status Unknown code -46
E0511 05:48:19.406781   121 BalanceTask.cpp:35] [1620708035, 1:2, test1-1-3-storaged-2.test1-1-3-storaged-headless.nebulacluster-5088.svc.cluster.local:9779->test1-1-3-storaged-3.test1-1-3-storaged-headless.nebulacluster-5088.svc.cluster.local:9779] Task failed, status 2
E0511 05:48:19.405524   129 BalanceTask.cpp:72] [1620708035, 1:3, test1-1-3-storaged-0.test1-1-3-storaged-headless.nebulacluster-5088.svc.cluster.local:9779->test1-1-3-storaged-3.test1-1-3-storaged-headless.nebulacluster-5088.svc.cluster.local:9779] Transfer leader failed, status Unknown code -46
E0511 05:48:19.406692   128 BalanceTask.cpp:35] [1620708035, 1:5, test1-1-3-storaged-2.test1-1-3-storaged-headless.nebulacluster-5088.svc.cluster.local:9779->test1-1-3-storaged-3.test1-1-3-storaged-headless.nebulacluster-5088.svc.cluster.local:9779] Task failed, status 2
E0511 05:48:19.405755   130 BalanceTask.cpp:72] [1620708035, 1:1, test1-1-3-storaged-1.test1-1-3-storaged-headless.nebulacluster-5088.svc.cluster.local:9779->test1-1-3-storaged-3.test1-1-3-storaged-headless.nebulacluster-5088.svc.cluster.local:9779] Transfer leader failed, status Unknown code -46
E0511 05:48:19.407047   129 BalanceTask.cpp:35] [1620708035, 1:3, test1-1-3-storaged-0.test1-1-3-storaged-headless.nebulacluster-5088.svc.cluster.local:9779->test1-1-3-storaged-3.test1-1-3-storaged-headless.nebulacluster-5088.svc.cluster.local:9779] Task failed, status 2
E0511 05:48:19.407377   130 BalanceTask.cpp:35] [1620708035, 1:1, test1-1-3-storaged-1.test1-1-3-storaged-headless.nebulacluster-5088.svc.cluster.local:9779->test1-1-3-storaged-3.test1-1-3-storaged-headless.nebulacluster-5088.svc.cluster.local:9779] Task failed, status 2

[critical27]

replica is 1 or 3?

[veezhang]

Nebula with 1 graph, 1 meta and 3 storage. And the replica_factor of space is 1.

[critical27]

👌

[darionyaphet]

seems TRANSFER_LEADER_FAILED... Because the replication number only one should not try to transfer it.

[critical27]

seems TRANSFER_LEADER_FAILED... Because the replication number only one should not try to transfer it.

It must be a bug, this issue is fixed in 1.0 before, balance for 1 replica should skip this phase in BalanceTask.

[critical27]

I probably know where does this problem comes from. Will fix, thx