Add service status as command option to admin command #4567

EdColeman · 2024-05-16T20:16:34Z

Adds a admin command serviceStatus to print service counts and hosts, with resource groups when appropriate to stdout.

To run the command:

> accumulo admin serviceStatus [--noHosts ] [--json]

The noHost option output provides a summary count
The --json option provides unformatted json output (the --noHost option is ignored)

Sample output:

Summary (service counts only)

> accumulo admin serviceStatus --noHosts
Report time: 2024-05-16T20:05:45.493Z
ZooKeeper read errors: 0
Managers: count: 1
Monitors: count: 1
Garbage Collectors: count: 1
Tablet Servers: count: 2
Scan Servers: count: 4
Coordinators: count: 1
Compactors: count: 4

Full status

> accumulo admin serviceStatus

Report time: 2024-05-16T20:04:09.556Z
ZooKeeper read errors: 0
Managers: count: 1
  localhost:9999
Monitors: count: 1
  localhost:9995
Garbage Collectors: count: 1
  localhost:9998
Tablet Servers: count: 2
  localhost:10000
  localhost:9997
Scan Servers: count: 4
  resource groups:
    default
    sg1
  hosts (by group):
    default: localhost:10003
    default: localhost:9996
    sg1: localhost:10004
    sg1: localhost:10005
Coordinators: count: 1
  localhost:9132
Compactors: count: 4
  resource groups:
    q1
    q2
  hosts (by group):
    q1: localhost:9133
    q1: localhost:9134
    q2: localhost:9135
    q2: localhost:9136

Full status formatted as json

> accumulo admin serviceStatus --json | jq   (normally the output is unformatted)

{
  "reportTime": "2024-05-16T20:01:39.593Z",
  "zkReadErrors": 0,
  "noHosts": false,
  "summaryMap": {
    "COMPACTOR": {
      "reportKey": "COMPACTOR",
      "resourceGroups": [
        "q1",
        "q2"
      ],
      "serviceNames": [
        "q1: localhost:9133",
        "q1: localhost:9134",
        "q2: localhost:9135",
        "q2: localhost:9136"
      ],
      "serviceCount": 4,
      "errorCount": 0
    },
    "COORDINATOR": {
      "reportKey": "COORDINATOR",
      "resourceGroups": [],
      "serviceNames": [
        "localhost:9132"
      ],
      "serviceCount": 1,
      "errorCount": 0
    },
    "GC": {
      "reportKey": "GC",
      "resourceGroups": [],
      "serviceNames": [
        "localhost:9998"
      ],
      "serviceCount": 1,
      "errorCount": 0
    },
    "MANAGER": {
      "reportKey": "MANAGER",
      "resourceGroups": [],
      "serviceNames": [
        "localhost:9999"
      ],
      "serviceCount": 1,
      "errorCount": 0
    },
    "MONITOR": {
      "reportKey": "MONITOR",
      "resourceGroups": [],
      "serviceNames": [
        "localhost:9995"
      ],
      "serviceCount": 1,
      "errorCount": 0
    },
    "S_SERVER": {
      "reportKey": "S_SERVER",
      "resourceGroups": [
        "default",
        "sg1"
      ],
      "serviceNames": [
        "default: localhost:10003",
        "default: localhost:9996",
        "sg1: localhost:10004",
        "sg1: localhost:10005"
      ],
      "serviceCount": 4,
      "errorCount": 0
    },
    "T_SERVER": {
      "reportKey": "T_SERVER",
      "resourceGroups": [],
      "serviceNames": [
        "localhost:10000",
        "localhost:9997"
      ],
      "serviceCount": 2,
      "errorCount": 0
    }
  }
}

Fixes #4495

dlmarion · 2024-05-17T16:02:25Z

server/base/src/main/java/org/apache/accumulo/server/util/ServiceStatusCmd.java

+          // process resource groups
+          var payload = r.getSecond();
+          String[] tokens = payload.split(",");
+          String groupSeparator = "";
+          if (tokens.length == 2) {
+            groupNames.add(tokens[1]);
+            groupSeparator = tokens[1] + ": ";
+          }


Is this parsing the lock content represented by the ServerServices object? If so, be aware that this changed in 3.1, ref: https://github.com/apache/accumulo/blob/main/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java#L664

dlmarion · 2024-05-17T16:07:24Z

...er/base/src/main/java/org/apache/accumulo/server/util/serviceStatus/ServiceStatusReport.java

+      sb.append(I2).append("resource groups:\n");
+      summary.getResourceGroups().forEach(g -> sb.append(I4).append(g).append("\n"));


Do we need to print the groups twice? I wonder if we could print something like:

Group: DEFAULT localhost:1234 localhost:1235 Group: Group1 localhost:1236

Address in f5f9221

This addresses PR comment to avoid displaying resource groups multiple times. It also changes the json output to use group name and a set of hosts. This makes it easier to pull out resources by resource group. Also includes some code clean-up

EdColeman · 2024-05-21T23:44:09Z

The output changes in f5f9221 organize services by resource group. This slightly modifies the command and the json output.

text output

> accumulo admin serviceStatus

Report time: 2024-05-21T23:06:21.084Z
ZooKeeper read errors: 0
Managers: count: 1
  localhost:9999
Monitors: count: 1
  localhost:9995
Garbage Collectors: count: 1
  localhost:9998
Tablet Servers: count: 2
  localhost:10000
  localhost:9997
Scan Servers: count: 4
  resource groups:
    default
    sg1
  hosts (by group):
    default (2):
      localhost:10003
      localhost:9996
    sg1 (2):
      localhost:10004
      localhost:10005
Coordinators: count: 1
  localhost:9132
Compactors: count: 4
  resource groups:
    q1
    q2
  hosts (by group):
    q1 (2):
      localhost:9133
      localhost:9134
    q2 (2):
      localhost:9135
      localhost:9136

json output (formatted with jq)

> accumulo admin serviceStatus --json | jq

{
  "reportTime": "2024-05-21T23:06:31.596Z",
  "zkReadErrors": 0,
  "noHosts": false,
  "summaries": {
    "COMPACTOR": {
      "serviceType": "COMPACTOR",
      "resourceGroups": [
        "q1",
        "q2"
      ],
      "serviceByGroups": {
        "q1": [
          "localhost:9133",
          "localhost:9134"
        ],
        "q2": [
          "localhost:9135",
          "localhost:9136"
        ]
      },
      "serviceCount": 4,
      "errorCount": 0
    },
    "COORDINATOR": {
      "serviceType": "COORDINATOR",
      "resourceGroups": [],
      "serviceByGroups": {
        "NO_GROUP": [
          "localhost:9132"
        ]
      },
      "serviceCount": 1,
      "errorCount": 0
    },
    "GC": {
      "serviceType": "GC",
      "resourceGroups": [],
      "serviceByGroups": {
        "NO_GROUP": [
          "localhost:9998"
        ]
      },
      "serviceCount": 1,
      "errorCount": 0
    },
    "MANAGER": {
      "serviceType": "MANAGER",
      "resourceGroups": [],
      "serviceByGroups": {
        "NO_GROUP": [
          "localhost:9999"
        ]
      },
      "serviceCount": 1,
      "errorCount": 0
    },
    "MONITOR": {
      "serviceType": "MONITOR",
      "resourceGroups": [],
      "serviceByGroups": {
        "NO_GROUP": [
          "localhost:9995"
        ]
      },
      "serviceCount": 1,
      "errorCount": 0
    },
    "S_SERVER": {
      "serviceType": "S_SERVER",
      "resourceGroups": [
        "default",
        "sg1"
      ],
      "serviceByGroups": {
        "default": [
          "localhost:10003",
          "localhost:9996"
        ],
        "sg1": [
          "localhost:10004",
          "localhost:10005"
        ]
      },
      "serviceCount": 4,
      "errorCount": 0
    },
    "T_SERVER": {
      "serviceType": "T_SERVER",
      "resourceGroups": [],
      "serviceByGroups": {
        "NO_GROUP": [
          "localhost:10000",
          "localhost:9997"
        ]
      },
      "serviceCount": 2,
      "errorCount": 0
    }
  }
}

dlmarion

LGTM. On the merge upstream, note the following:

The classs ServiceLockData can be used to parse the lock data in main
The Coordinator was merged into the Manager in elasticity

- the merge of #4567 needs additional changes because of ServiceLock data in ZooKeeper. This will be done as a separate PR.

Add service status as command option to admin command

dc96ce2

EdColeman requested a review from dlmarion May 16, 2024 20:17

EdColeman mentioned this pull request May 16, 2024

Create an accumulo status command #4495

Closed

dlmarion reviewed May 17, 2024

View reviewed changes

Organize service status by resource group.

f5f9221

This addresses PR comment to avoid displaying resource groups multiple times. It also changes the json output to use group name and a set of hosts. This makes it easier to pull out resources by resource group. Also includes some code clean-up

Add report getters and json round trip test

647a6cb

dlmarion approved these changes May 24, 2024

View reviewed changes

Ed Coleman added 3 commits May 24, 2024 15:25

Merge remote-tracking branch 'upstream/2.1' into admin_status_command

f46ee12

Merge remote-tracking branch 'upstream/2.1' into admin_status_command

1d12b0a

Merge remote-tracking branch 'upstream/2.1' into admin_status_command

ae7875e

EdColeman merged commit e3c540d into apache:2.1 May 24, 2024
8 checks passed

EdColeman pushed a commit that referenced this pull request May 24, 2024

Merge remote-tracking branch 'upstream/2.1'

8e5d967

- the merge of #4567 needs additional changes because of ServiceLock data in ZooKeeper. This will be done as a separate PR.

keith-turner mentioned this pull request May 28, 2024

Service status command is not functional in elasticity after merge #4612

Closed

ctubbsii modified the milestones: 4.0.0, 3.1.0, 2.1.3 Jul 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add service status as command option to admin command #4567

Add service status as command option to admin command #4567

EdColeman commented May 16, 2024

dlmarion May 17, 2024

dlmarion May 17, 2024

EdColeman May 21, 2024

EdColeman commented May 21, 2024

dlmarion left a comment

		sb.append(I2).append("resource groups:\n");
		summary.getResourceGroups().forEach(g -> sb.append(I4).append(g).append("\n"));

Add service status as command option to admin command #4567

Add service status as command option to admin command #4567

Conversation

EdColeman commented May 16, 2024

dlmarion May 17, 2024

Choose a reason for hiding this comment

dlmarion May 17, 2024

Choose a reason for hiding this comment

EdColeman May 21, 2024

Choose a reason for hiding this comment

EdColeman commented May 21, 2024

dlmarion left a comment

Choose a reason for hiding this comment