Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

balance data with zone, 2 part peers in the same zone #3029

Closed
HarrisChu opened this issue Oct 11, 2021 · 2 comments · Fixed by #3059
Closed

balance data with zone, 2 part peers in the same zone #3029

HarrisChu opened this issue Oct 11, 2021 · 2 comments · Fixed by #3059
Assignees
@darionyaphet
Labels
priority/hi-pri Priority: high type/bug Type: something is unexpected
Milestone
v2.6.0

Comments

@HarrisChu
Copy link
Contributor

steps:

  1. create space
  2. balance data

expect:

  1. 1 part peer in one zone.

actual:

  1. 2 part peers in one zone.
(root@nebula) [s3]> show zones
+------+------------------------------------------------------------------------+------+
| Name | Host                                                                   | Port |
+------+------------------------------------------------------------------------+------+
| "z1" | "cbrpnm-storaged-0.cbrpnm-storaged-headless.default.svc.cluster.local" | 9779 |
+------+------------------------------------------------------------------------+------+
| "z1" | "cbrpnm-storaged-1.cbrpnm-storaged-headless.default.svc.cluster.local" | 9779 |
+------+------------------------------------------------------------------------+------+
| "z1" | "cbrpnm-storaged-2.cbrpnm-storaged-headless.default.svc.cluster.local" | 9779 |
+------+------------------------------------------------------------------------+------+
| "z2" | "cbrpnm-storaged-3.cbrpnm-storaged-headless.default.svc.cluster.local" | 9779 |
+------+------------------------------------------------------------------------+------+
| "z3" | "cbrpnm-storaged-4.cbrpnm-storaged-headless.default.svc.cluster.local" | 9779 |
+------+------------------------------------------------------------------------+------+
| "z4" | "cbrpnm-storaged-5.cbrpnm-storaged-headless.default.svc.cluster.local" | 9779 |
+------+------------------------------------------------------------------------+------+
| "z5" | "cbrpnm-storaged-6.cbrpnm-storaged-headless.default.svc.cluster.local" | 9779 |
+------+------------------------------------------------------------------------+------+
Got 7 rows (time spent 941/8731 us)

Mon, 11 Oct 2021 11:24:33 CST

(root@nebula) [s3]> show groups
+------+------+
| Name | Zone |
+------+------+
| "g1" | "z1" |
+------+------+
| "g1" | "z2" |
+------+------+
| "g1" | "z3" |
+------+------+
| "g1" | "z4" |
+------+------+
| "g1" | "z5" |
+------+------+
Got 5 rows (time spent 753/10334 us)

Mon, 11 Oct 2021 11:24:36 CST

(root@nebula) [s3]> create space s5(replica_factor=3, vid_type=int, partition_num=8) on g1
Execution succeeded (time spent 1003/8995 us)

Mon, 11 Oct 2021 11:24:48 CST

(root@nebula) [s5]> show parts
+--------------+-----------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| Partition ID | Leader                                                                      | Peers                                                                                                                                                                                                                             | Losts |
+--------------+-----------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| 1            | "cbrpnm-storaged-5.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | "cbrpnm-storaged-5.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-6.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-4.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | ""    |
+--------------+-----------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| 2            | "cbrpnm-storaged-4.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | "cbrpnm-storaged-5.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-6.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-4.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | ""    |
+--------------+-----------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| 3            | "cbrpnm-storaged-6.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | "cbrpnm-storaged-5.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-6.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-4.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | ""    |
+--------------+-----------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| 4            | "cbrpnm-storaged-6.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | "cbrpnm-storaged-5.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-6.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-3.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | ""    |
+--------------+-----------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| 5            | "cbrpnm-storaged-4.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | "cbrpnm-storaged-5.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-4.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-3.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | ""    |
+--------------+-----------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| 6            | "cbrpnm-storaged-3.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | "cbrpnm-storaged-6.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-4.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-3.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | ""    |
+--------------+-----------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| 7            | "cbrpnm-storaged-4.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | "cbrpnm-storaged-5.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-6.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-4.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | ""    |
+--------------+-----------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| 8            | "cbrpnm-storaged-6.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | "cbrpnm-storaged-5.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-6.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-3.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | ""    |
+--------------+-----------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
Got 8 rows (time spent 785/8811 us)

Mon, 11 Oct 2021 11:24:53 CST

(root@nebula) [s5]> balance data
+------------+
| ID         |
+------------+
| 1633922715 |
+------------+
Got 1 rows (time spent 2734/11429 us)

Mon, 11 Oct 2021 11:25:10 CST

(root@nebula) [s5]> balance data 1633922715
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+
| balanceId, spaceId:partId, src->dst                                                                                                                                        | status            |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+
| "[1633922715, 11:1, cbrpnm-storaged-4.cbrpnm-storaged-headless.default.svc.cluster.local:9779->cbrpnm-storaged-0.cbrpnm-storaged-headless.default.svc.cluster.local:9779]" | "IN_PROGRESS"     |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+
| "[1633922715, 11:1, cbrpnm-storaged-5.cbrpnm-storaged-headless.default.svc.cluster.local:9779->cbrpnm-storaged-1.cbrpnm-storaged-headless.default.svc.cluster.local:9779]" | "IN_PROGRESS"     |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+
| "[1633922715, 11:1, cbrpnm-storaged-6.cbrpnm-storaged-headless.default.svc.cluster.local:9779->cbrpnm-storaged-2.cbrpnm-storaged-headless.default.svc.cluster.local:9779]" | "IN_PROGRESS"     |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+
| "[1633922715, 11:2, cbrpnm-storaged-4.cbrpnm-storaged-headless.default.svc.cluster.local:9779->cbrpnm-storaged-0.cbrpnm-storaged-headless.default.svc.cluster.local:9779]" | "IN_PROGRESS"     |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+
| "[1633922715, 11:2, cbrpnm-storaged-5.cbrpnm-storaged-headless.default.svc.cluster.local:9779->cbrpnm-storaged-1.cbrpnm-storaged-headless.default.svc.cluster.local:9779]" | "IN_PROGRESS"     |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+
| "[1633922715, 11:2, cbrpnm-storaged-6.cbrpnm-storaged-headless.default.svc.cluster.local:9779->cbrpnm-storaged-2.cbrpnm-storaged-headless.default.svc.cluster.local:9779]" | "IN_PROGRESS"     |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+
| "[1633922715, 11:3, cbrpnm-storaged-4.cbrpnm-storaged-headless.default.svc.cluster.local:9779->cbrpnm-storaged-0.cbrpnm-storaged-headless.default.svc.cluster.local:9779]" | "SUCCEEDED"       |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+
| "[1633922715, 11:3, cbrpnm-storaged-5.cbrpnm-storaged-headless.default.svc.cluster.local:9779->cbrpnm-storaged-1.cbrpnm-storaged-headless.default.svc.cluster.local:9779]" | "IN_PROGRESS"     |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+
| "[1633922715, 11:3, cbrpnm-storaged-6.cbrpnm-storaged-headless.default.svc.cluster.local:9779->cbrpnm-storaged-2.cbrpnm-storaged-headless.default.svc.cluster.local:9779]" | "IN_PROGRESS"     |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+
| "Total:9, Succeeded:1, Failed:0, In Progress:8, Invalid:0"                                                                                                                 | 11.11111111111111 |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+
Got 10 rows (time spent 912/6524 us)

Mon, 11 Oct 2021 11:25:16 CST

(root@nebula) [s5]> show parts
+--------------+-----------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| Partition ID | Leader                                                                      | Peers                                                                                                                                                                                                                             | Losts |
+--------------+-----------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| 1            | "cbrpnm-storaged-0.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | "cbrpnm-storaged-0.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-1.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-2.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | ""    |
+--------------+-----------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| 2            | "cbrpnm-storaged-0.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | "cbrpnm-storaged-0.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-1.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-2.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | ""    |
+--------------+-----------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| 3            | "cbrpnm-storaged-0.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | "cbrpnm-storaged-0.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-1.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-2.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | ""    |
+--------------+-----------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| 4            | "cbrpnm-storaged-6.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | "cbrpnm-storaged-5.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-6.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-3.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | ""    |
+--------------+-----------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| 5            | "cbrpnm-storaged-4.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | "cbrpnm-storaged-5.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-4.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-3.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | ""    |
+--------------+-----------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| 6            | "cbrpnm-storaged-3.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | "cbrpnm-storaged-6.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-4.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-3.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | ""    |
+--------------+-----------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| 7            | "cbrpnm-storaged-4.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | "cbrpnm-storaged-5.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-6.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-4.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | ""    |
+--------------+-----------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
| 8            | "cbrpnm-storaged-6.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | "cbrpnm-storaged-5.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-6.cbrpnm-storaged-headless.default.svc.cluster.local:9779, cbrpnm-storaged-3.cbrpnm-storaged-headless.default.svc.cluster.local:9779" | ""    |
+--------------+-----------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
Got 8 rows (time spent 771/4791 us)

Mon, 11 Oct 2021 11:26:51 CST
@HarrisChu HarrisChu added the type/bug Type: something is unexpected label Oct 11, 2021
@HarrisChu
Copy link
Contributor Author 

(root@nebula) [s5]> balance data 1633922715
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+
| balanceId, spaceId:partId, src->dst                                                                                                                                        | status            |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+
| "[1633922715, 11:1, cbrpnm-storaged-4.cbrpnm-storaged-headless.default.svc.cluster.local:9779->cbrpnm-storaged-0.cbrpnm-storaged-headless.default.svc.cluster.local:9779]" | "IN_PROGRESS"     |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+
| "[1633922715, 11:1, cbrpnm-storaged-5.cbrpnm-storaged-headless.default.svc.cluster.local:9779->cbrpnm-storaged-1.cbrpnm-storaged-headless.default.svc.cluster.local:9779]" | "IN_PROGRESS"     |

balance plan is wrong, cbrpnm-storaged-0.cbrpnm-storaged-headless.default.svc.cluster.local:9779 and cbrpnm-storaged-1.cbrpnm-storaged-headless.default.svc.cluster.local:9779 are in the same zone.

@HarrisChu HarrisChu added the priority/hi-pri Priority: high label Oct 11, 2021
@HarrisChu HarrisChu added this to the v2.6.0 milestone Oct 11, 2021
@darionyaphet
Copy link
Contributor

#3059

This was linked to pull requests Oct 14, 2021
Cascading deletion and addition parts distribution #3059
Merged
fix part peers in the same zone #3047
Closed
@Sophie-Xie Sophie-Xie removed a link to a pull request Oct 14, 2021
fix part peers in the same zone #3047
Closed
@jamieliu1023 jamieliu1023 mentioned this issue Oct 20, 2021
Closed
@jamieliu1023 jamieliu1023 mentioned this issue Nov 1, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@darionyaphet darionyaphet

Labels
priority/hi-pri Priority: high type/bug Type: something is unexpected
Projects
None yet
Milestone
v2.6.0
Development

Successfully merging a pull request may close this issue.

Cascading deletion and addition parts distribution darionyaphet/nebula
2 participants
@HarrisChu @darionyaphet