Support "show parts" command #1086

zhangguoqing · 2019-10-15T14:02:11Z

Add the ListPartsProcessor to get all partitions' information(id, leader, peers, losts) of one graph space.
The SHOW PARTS command can show the results displayed by console, after doing USE space_name.

close #930

Example:

Init and start the nebula graph cluster:

(user@172.25.61.1) [(none)]> show hosts;
==============================================================================================
| Ip          | Port  | Status | Leader count | Leader distribution | Partition distribution |
==============================================================================================
| 172.25.61.1 | 44600 | online | 0            | No valid partition  | No valid partition     |
----------------------------------------------------------------------------------------------
| 172.25.61.1 | 44700 | online | 0            | No valid partition  | No valid partition     |
----------------------------------------------------------------------------------------------
| 172.25.61.1 | 44500 | online | 0            | No valid partition  | No valid partition     |
----------------------------------------------------------------------------------------------

Create the graph spaces(named myspace) with 9 partitions and 3 replicas:

(user@172.25.61.1) [(none)]> CREATE SPACE myspace(partition_num=9, replica_factor=3);
Execution succeeded (Time spent: 5845/6800 us)

Show the hosts with myspace:

(user@172.25.61.1) [(none)]> show hosts;
==============================================================================================
| Ip          | Port  | Status | Leader count | Leader distribution | Partition distribution |
==============================================================================================
| 172.25.61.1 | 44600 | online | 0            | myspace: 0          | myspace: 9             |
----------------------------------------------------------------------------------------------
| 172.25.61.1 | 44700 | online | 0            | myspace: 0          | myspace: 9             |
----------------------------------------------------------------------------------------------
| 172.25.61.1 | 44500 | online | 0            | myspace: 0          | myspace: 9             |
----------------------------------------------------------------------------------------------

Show parts of myspace, before leader election:

(user@172.25.61.1) [(none)]> use myspace;
Execution succeeded (Time spent: 1463/2378 us)

(user@172.25.61.1) [myspace]> show parts;
===========================================================================================
| Partition ID | Leader | Peers                                                   | Losts |
===========================================================================================
| 1            |        | 172.25.61.1:44700, 172.25.61.1:44500, 172.25.61.1:44600 |       |
-------------------------------------------------------------------------------------------
| 2            |        | 172.25.61.1:44500, 172.25.61.1:44600, 172.25.61.1:44700 |       |
-------------------------------------------------------------------------------------------
| 3            |        | 172.25.61.1:44600, 172.25.61.1:44700, 172.25.61.1:44500 |       |
-------------------------------------------------------------------------------------------
| 4            |        | 172.25.61.1:44700, 172.25.61.1:44500, 172.25.61.1:44600 |       |
-------------------------------------------------------------------------------------------
| 5            |        | 172.25.61.1:44500, 172.25.61.1:44600, 172.25.61.1:44700 |       |
-------------------------------------------------------------------------------------------
| 6            |        | 172.25.61.1:44600, 172.25.61.1:44700, 172.25.61.1:44500 |       |
-------------------------------------------------------------------------------------------
| 7            |        | 172.25.61.1:44700, 172.25.61.1:44500, 172.25.61.1:44600 |       |
-------------------------------------------------------------------------------------------
| 8            |        | 172.25.61.1:44500, 172.25.61.1:44600, 172.25.61.1:44700 |       |
-------------------------------------------------------------------------------------------
| 9            |        | 172.25.61.1:44600, 172.25.61.1:44700, 172.25.61.1:44500 |       |
-------------------------------------------------------------------------------------------

Show parts of myspace, after leader election:

(user@172.25.61.1) [myspace]> show parts;
======================================================================================================
| Partition ID | Leader            | Peers                                                   | Losts |
======================================================================================================
| 1            | 172.25.61.1:44500 | 172.25.61.1:44700, 172.25.61.1:44500, 172.25.61.1:44600 |       |
------------------------------------------------------------------------------------------------------
| 2            | 172.25.61.1:44500 | 172.25.61.1:44500, 172.25.61.1:44600, 172.25.61.1:44700 |       |
------------------------------------------------------------------------------------------------------
| 3            | 172.25.61.1:44500 | 172.25.61.1:44600, 172.25.61.1:44700, 172.25.61.1:44500 |       |
------------------------------------------------------------------------------------------------------
| 4            | 172.25.61.1:44500 | 172.25.61.1:44700, 172.25.61.1:44500, 172.25.61.1:44600 |       |
------------------------------------------------------------------------------------------------------
| 5            | 172.25.61.1:44500 | 172.25.61.1:44500, 172.25.61.1:44600, 172.25.61.1:44700 |       |
------------------------------------------------------------------------------------------------------
| 6            | 172.25.61.1:44500 | 172.25.61.1:44600, 172.25.61.1:44700, 172.25.61.1:44500 |       |
------------------------------------------------------------------------------------------------------
| 7            | 172.25.61.1:44500 | 172.25.61.1:44700, 172.25.61.1:44500, 172.25.61.1:44600 |       |
------------------------------------------------------------------------------------------------------
| 8            | 172.25.61.1:44500 | 172.25.61.1:44500, 172.25.61.1:44600, 172.25.61.1:44700 |       |
------------------------------------------------------------------------------------------------------
| 9            | 172.25.61.1:44500 | 172.25.61.1:44600, 172.25.61.1:44700, 172.25.61.1:44500 |       |
------------------------------------------------------------------------------------------------------

Stop the storaged 172.25.61.1:44500:

(user@172.25.61.1) [myspace]> show hosts;
==============================================================================================
| Ip          | Port  | Status | Leader count | Leader distribution | Partition distribution |
==============================================================================================
| 172.25.61.1 | 44600 | online | 7            | myspace: 7          | myspace: 9             |
----------------------------------------------------------------------------------------------
| 172.25.61.1 | 44700 | online | 2            | myspace: 2          | myspace: 9             |
----------------------------------------------------------------------------------------------
| 172.25.61.1 | 44500 | online | 0            | myspace: 0          | myspace: 9             |
----------------------------------------------------------------------------------------------
Got 3 rows (Time spent: 155542/156820 us)

(user@172.25.61.1) [myspace]> show parts;
======================================================================================================
| Partition ID | Leader            | Peers                                                   | Losts |
======================================================================================================
| 1            | 172.25.61.1:44600 | 172.25.61.1:44700, 172.25.61.1:44500, 172.25.61.1:44600 |       |
------------------------------------------------------------------------------------------------------
| 2            | 172.25.61.1:44600 | 172.25.61.1:44500, 172.25.61.1:44600, 172.25.61.1:44700 |       |
------------------------------------------------------------------------------------------------------
| 3            | 172.25.61.1:44600 | 172.25.61.1:44600, 172.25.61.1:44700, 172.25.61.1:44500 |       |
------------------------------------------------------------------------------------------------------
| 4            | 172.25.61.1:44600 | 172.25.61.1:44700, 172.25.61.1:44500, 172.25.61.1:44600 |       |
------------------------------------------------------------------------------------------------------
| 5            | 172.25.61.1:44600 | 172.25.61.1:44500, 172.25.61.1:44600, 172.25.61.1:44700 |       |
------------------------------------------------------------------------------------------------------
| 6            | 172.25.61.1:44700 | 172.25.61.1:44600, 172.25.61.1:44700, 172.25.61.1:44500 |       |
------------------------------------------------------------------------------------------------------
| 7            | 172.25.61.1:44700 | 172.25.61.1:44700, 172.25.61.1:44500, 172.25.61.1:44600 |       |
------------------------------------------------------------------------------------------------------
| 8            | 172.25.61.1:44600 | 172.25.61.1:44500, 172.25.61.1:44600, 172.25.61.1:44700 |       |
------------------------------------------------------------------------------------------------------
| 9            | 172.25.61.1:44600 | 172.25.61.1:44600, 172.25.61.1:44700, 172.25.61.1:44500 |       |
------------------------------------------------------------------------------------------------------

Wait for the expired_threshold_sec(10 * 60) seconds:

(user@172.25.61.1) [myspace]> show hosts;
===============================================================================================
| Ip          | Port  | Status  | Leader count | Leader distribution | Partition distribution |
===============================================================================================
| 172.25.61.1 | 44600 | online  | 7            | myspace: 7          | myspace: 9             |
-----------------------------------------------------------------------------------------------
| 172.25.61.1 | 44700 | online  | 2            | myspace: 2          | myspace: 9             |
-----------------------------------------------------------------------------------------------
| 172.25.61.1 | 44500 | offline | 0            | myspace: 0          | myspace: 9             |
-----------------------------------------------------------------------------------------------
Got 3 rows (Time spent: 2928/3980 us)

(user@172.25.61.1) [myspace]> show parts;
==================================================================================================================
| Partition ID | Leader            | Peers                                                   | Losts             |
==================================================================================================================
| 1            | 172.25.61.1:44600 | 172.25.61.1:44700, 172.25.61.1:44500, 172.25.61.1:44600 | 172.25.61.1:44500 |
------------------------------------------------------------------------------------------------------------------
| 2            | 172.25.61.1:44600 | 172.25.61.1:44500, 172.25.61.1:44600, 172.25.61.1:44700 | 172.25.61.1:44500 |
------------------------------------------------------------------------------------------------------------------
| 3            | 172.25.61.1:44600 | 172.25.61.1:44600, 172.25.61.1:44700, 172.25.61.1:44500 | 172.25.61.1:44500 |
------------------------------------------------------------------------------------------------------------------
| 4            | 172.25.61.1:44600 | 172.25.61.1:44700, 172.25.61.1:44500, 172.25.61.1:44600 | 172.25.61.1:44500 |
------------------------------------------------------------------------------------------------------------------
| 5            | 172.25.61.1:44600 | 172.25.61.1:44500, 172.25.61.1:44600, 172.25.61.1:44700 | 172.25.61.1:44500 |
------------------------------------------------------------------------------------------------------------------
| 6            | 172.25.61.1:44700 | 172.25.61.1:44600, 172.25.61.1:44700, 172.25.61.1:44500 | 172.25.61.1:44500 |
------------------------------------------------------------------------------------------------------------------
| 7            | 172.25.61.1:44700 | 172.25.61.1:44700, 172.25.61.1:44500, 172.25.61.1:44600 | 172.25.61.1:44500 |
------------------------------------------------------------------------------------------------------------------
| 8            | 172.25.61.1:44600 | 172.25.61.1:44500, 172.25.61.1:44600, 172.25.61.1:44700 | 172.25.61.1:44500 |
------------------------------------------------------------------------------------------------------------------
| 9            | 172.25.61.1:44600 | 172.25.61.1:44600, 172.25.61.1:44700, 172.25.61.1:44500 | 172.25.61.1:44500 |
------------------------------------------------------------------------------------------------------------------

Start the storaged 172.25.61.1:44500 again:

[root@YZ-25-61-1 nebula_install1]# ./scripts/nebula-storaged.service start
[INFO] Starting nebula-storaged...
[INFO] Done

[root@YZ-25-61-1 nebula_install1]# bin/nebula --addr=172.25.61.1  --port=8000  --u=user --p=password

Welcome to Nebula Graph

(user@172.25.61.1) [(none)]> show hosts;
==============================================================================================
| Ip          | Port  | Status | Leader count | Leader distribution | Partition distribution |
==============================================================================================
| 172.25.61.1 | 44600 | online | 7            | myspace: 7          | myspace: 9             |
----------------------------------------------------------------------------------------------
| 172.25.61.1 | 44700 | online | 2            | myspace: 2          | myspace: 9             |
----------------------------------------------------------------------------------------------
| 172.25.61.1 | 44500 | online | 0            | myspace: 0          | myspace: 9             |
----------------------------------------------------------------------------------------------
Got 3 rows (Time spent: 3776/4860 us)

(user@172.25.61.1) [(none)]> use myspace;
Execution succeeded (Time spent: 1473/2444 us)

(user@172.25.61.1) [myspace]> show parts;
======================================================================================================
| Partition ID | Leader            | Peers                                                   | Losts |
======================================================================================================
| 1            | 172.25.61.1:44600 | 172.25.61.1:44700, 172.25.61.1:44500, 172.25.61.1:44600 |       |
------------------------------------------------------------------------------------------------------
| 2            | 172.25.61.1:44600 | 172.25.61.1:44500, 172.25.61.1:44600, 172.25.61.1:44700 |       |
------------------------------------------------------------------------------------------------------
| 3            | 172.25.61.1:44600 | 172.25.61.1:44600, 172.25.61.1:44700, 172.25.61.1:44500 |       |
------------------------------------------------------------------------------------------------------
| 4            | 172.25.61.1:44600 | 172.25.61.1:44700, 172.25.61.1:44500, 172.25.61.1:44600 |       |
------------------------------------------------------------------------------------------------------
| 5            | 172.25.61.1:44600 | 172.25.61.1:44500, 172.25.61.1:44600, 172.25.61.1:44700 |       |
------------------------------------------------------------------------------------------------------
| 6            | 172.25.61.1:44700 | 172.25.61.1:44600, 172.25.61.1:44700, 172.25.61.1:44500 |       |
------------------------------------------------------------------------------------------------------
| 7            | 172.25.61.1:44700 | 172.25.61.1:44700, 172.25.61.1:44500, 172.25.61.1:44600 |       |
------------------------------------------------------------------------------------------------------
| 8            | 172.25.61.1:44600 | 172.25.61.1:44500, 172.25.61.1:44600, 172.25.61.1:44700 |       |
------------------------------------------------------------------------------------------------------
| 9            | 172.25.61.1:44600 | 172.25.61.1:44600, 172.25.61.1:44700, 172.25.61.1:44500 |       |
------------------------------------------------------------------------------------------------------

nebula-community-bot · 2019-10-17T13:12:49Z

Unit testing failed.

dangleptr

That's cool. Could we show the results displayed by console?

dangleptr · 2019-10-29T03:55:52Z

Jenkins go

nebula-community-bot · 2019-10-29T04:18:08Z

Unit testing failed.

zhangguoqing · 2019-10-29T06:08:57Z

That's cool. Could we show the results displayed by console?

of course!

zhangguoqing · 2019-10-29T09:03:58Z

Jenkins go

nebula-community-bot · 2019-10-29T09:26:21Z

Unit testing passed.

nebula-community-bot · 2019-10-29T09:28:47Z

Unit testing passed.

zhangguoqing · 2019-10-29T11:09:25Z

Jenkins go

nebula-community-bot · 2019-10-29T11:24:44Z

Unit testing passed.

nebula-community-bot · 2019-10-29T11:27:08Z

Unit testing passed.

dangleptr · 2019-10-30T02:17:31Z

That's cool. Could we show the results displayed by console?

of course!

You could post it in the summary.

zhangguoqing · 2019-10-30T03:25:19Z

That's cool. Could we show the results displayed by console?

of course!

You could post it in the summary.

Thanks, I have updated the summary of this PR.

dangleptr · 2019-10-30T04:25:39Z

That's cool. Could we show the results displayed by console?

of course!

You could post it in the summary.

Thanks, I have updated the summary of this PR.

I means you could post the results displayed by console in this summary.

zhangguoqing · 2019-10-30T07:12:31Z

That's cool. Could we show the results displayed by console?

of course!

You could post it in the summary.

Thanks, I have updated the summary of this PR.

I means you could post the results displayed by console in this summary.

Done!

dangleptr

Well done!

dangleptr · 2019-10-30T08:37:58Z

src/common/network/NetworkUtils.cpp

+        std::string addrStr = network::NetworkUtils::ipFromHostAddr(host);
+        int32_t port = network::NetworkUtils::portFromHostAddr(host);
+        std::string portStr = folly::to<std::string>(port);
+        hostsString += addrStr + ":" + portStr + ", ";


folly::stringPrintf("%s:%d,", addrStr.c_str(), port);

nebula-community-bot · 2019-10-30T12:02:06Z

Unit testing passed.

nebula-community-bot · 2019-11-01T11:32:27Z

Unit testing failed.

critical27 · 2019-11-04T02:29:43Z

src/meta/processors/partsMan/ListPartsProcessor.cpp

+namespace nebula {
+namespace meta {
+
+void ListPartsProcessor::process(const cpp2::ListPartsReq& req) {


Do you think we can support list the partition specified in console? If we have to many partitions, it's hard to look for a single partition.

Agreed. We'd better support to show some part information specified too.

The result is sorted by partition ID.

Of course, we could open another pr for it.

Yep, would it be better to use the LIMIT clause?

close vesoft-inc#930

critical27

Clean code! LGTM

zhangguoqing · 2019-11-05T02:07:42Z

Jenkins go!

liuyu85cn · 2019-11-05T02:30:47Z

src/common/network/NetworkUtils.cpp

@@ -306,6 +306,19 @@ StatusOr<std::vector<HostAddr>> NetworkUtils::toHosts(const std::string& peersSt
    return hosts;
 }

+std::string NetworkUtils::toHosts(const std::vector<HostAddr>& hosts) {


concatHosts may be a better func name to explain this?

Thx. I prefer the toHosts function overload.

concatHosts +1

IMO, the Concat generally refers to joining the same kind of things and returning the original form. For example, joining multiple strings returns a string. There is a formal shift here(HostAddr to String, with the meaning of convert to), not a simple join or concat.

nebula-community-bot · 2019-11-05T02:30:56Z

Unit testing passed.

nebula-community-bot · 2019-11-05T02:33:37Z

Unit testing passed.

nebula-community-bot · 2019-11-05T03:02:27Z

Unit testing passed.

nebula-community-bot · 2019-11-07T05:37:23Z

Unit testing passed.

dangleptr

LGTM

jude-zhu · 2019-11-28T06:22:53Z

subtask of #724

close vesoft-inc#930

zhangguoqing force-pushed the show-parts branch 2 times, most recently from 7102a05 to aeeb9ae Compare October 17, 2019 03:58

zhangguoqing changed the title ~~[WIP] Support "show parts" command~~ Support "show parts" command Oct 17, 2019

zhangguoqing force-pushed the show-parts branch from aeeb9ae to acbbc2c Compare October 17, 2019 08:03

zhangguoqing requested a review from dangleptr October 17, 2019 08:03

zhangguoqing force-pushed the show-parts branch from acbbc2c to 4428029 Compare October 17, 2019 12:48

zhangguoqing added the ready-for-testing PR: ready for the CI test label Oct 17, 2019

dangleptr reviewed Oct 29, 2019

View reviewed changes

zhangguoqing force-pushed the show-parts branch from 4428029 to 1a3530a Compare October 29, 2019 09:01

zhangguoqing force-pushed the show-parts branch from 1a3530a to 243d67a Compare October 29, 2019 11:08

critical27 mentioned this pull request Oct 30, 2019

Improve heartbeat of storage. #1173

Closed

zhangguoqing requested review from critical27, CPWstatic, darionyaphet, laura-ding and zlcook October 30, 2019 07:42

dangleptr previously approved these changes Oct 30, 2019

View reviewed changes

dangleptr reviewed Oct 30, 2019

View reviewed changes

critical27 reviewed Nov 4, 2019

View reviewed changes

Support "show parts" command

ef9ea77

close vesoft-inc#930

zhangguoqing dismissed dangleptr’s stale review via ef9ea77 November 5, 2019 01:59

zhangguoqing force-pushed the show-parts branch from 2a8f708 to ef9ea77 Compare November 5, 2019 01:59

critical27 approved these changes Nov 5, 2019

View reviewed changes

liuyu85cn reviewed Nov 5, 2019

View reviewed changes

Merge branch 'master' into show-parts

2f190dc

dangleptr approved these changes Nov 7, 2019

View reviewed changes

dangleptr merged commit e0e2530 into vesoft-inc:master Nov 7, 2019

zhangguoqing mentioned this pull request Nov 7, 2019

Support list the partition specified in console #1226

Closed

yixinglu pushed a commit to yixinglu/nebula that referenced this pull request Feb 16, 2020

Support "show parts" command (vesoft-inc#1086)

9922295

close vesoft-inc#930

tong-hao pushed a commit to tong-hao/nebula that referenced this pull request Jun 1, 2021

Support "show parts" command (vesoft-inc#1086)

c23e3fe

close vesoft-inc#930

Support "show parts" command #1086

Support "show parts" command #1086

Conversation

zhangguoqing commented Oct 15, 2019 • edited by dangleptr Loading

nebula-community-bot commented Oct 17, 2019

dangleptr left a comment

Choose a reason for hiding this comment

dangleptr commented Oct 29, 2019

nebula-community-bot commented Oct 29, 2019

zhangguoqing commented Oct 29, 2019

zhangguoqing commented Oct 29, 2019

nebula-community-bot commented Oct 29, 2019

nebula-community-bot commented Oct 29, 2019

zhangguoqing commented Oct 29, 2019

nebula-community-bot commented Oct 29, 2019

nebula-community-bot commented Oct 29, 2019

dangleptr commented Oct 30, 2019

zhangguoqing commented Oct 30, 2019

dangleptr commented Oct 30, 2019

zhangguoqing commented Oct 30, 2019 • edited Loading

dangleptr left a comment

Choose a reason for hiding this comment

dangleptr Oct 30, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nebula-community-bot commented Oct 30, 2019

nebula-community-bot commented Nov 1, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

critical27 left a comment

Choose a reason for hiding this comment

zhangguoqing commented Nov 5, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nebula-community-bot commented Nov 5, 2019

nebula-community-bot commented Nov 5, 2019

nebula-community-bot commented Nov 5, 2019

nebula-community-bot commented Nov 7, 2019

dangleptr left a comment

Choose a reason for hiding this comment

jude-zhu commented Nov 28, 2019

zhangguoqing commented Oct 15, 2019 •

edited by dangleptr

Loading

zhangguoqing commented Oct 30, 2019 •

edited

Loading

dangleptr Oct 30, 2019 •

edited

Loading