Elasticsearch Stack Monitoring: shards data

[ycombinator]

Taken from https://docs.google.com/document/d/1FtGp_eoecFg4Q3xKeOn_Hi5yx3Evz2qeDxsk49g6Qpc/edit# since @chrisronline did such an excellent job with the write up there.

We show shard-related data in a few places in the Stack Monitoring UI, specifically the index and node detail pages. Shard data lives in separate monitoring documents with a specific type shards. Index and node data lives also live in separate monitoring documents with specific types: index_stats and node_stats. Currently, we are querying both sets of documents to get all the data necessary to render the page (For example, on the node detail page, we are querying node_stats and shards). This isn’t ideal because:

• We need to make serial requests to get all the data we need
• We store additional, unnecessary documents containing shard data - We never query just this data for any UI
• We need to worry about unique-ness with shards, see this and this

We recently made changes to improve this, but it feels like we need to change this for a scalable solution.

As of right now, this is the data contained in a type “shards” document:

"shard" : {
  "node" : "CNdjzk_dRaGURxVJ3_c9Xw",
  "index" : ".monitoring-logstash-6-2017.08.23",
  "shard" : 0,
  "state" : "STARTED",
  "primary" : true
},

The index_stats monitoring documents currently contain some shard data:

"shards" : {
  "replicas" : 0,
  "unassigned_primaries" : 0,
  "initializing" : 0,
  "primaries" : 1,
  "total" : 1,
  "active_total" : 1,
  "unassigned_replicas" : 0,
  "active_primaries" : 1,
  "active_replicas" : 0,
  "unassigned_total" : 0,
  "relocating" : 0
}

This is a great start, but it’s missing some data and contains data that isn’t used.

---

I’m proposing a breaking change for 8.0 that involves adding data to three existing monitoring document types and completely removing the type “shards” document.

The additional data:

For “node_stats”:

“node_stats.indexCount”: <number>,
“node_stats.shards”: [
  {
    “index”: <number>,
    “state”: <string>,
    “primary”: <boolean>
  }
]

For “index_stats”:

“index_stats.shards”: [
  {
    “shard_index”: <number>,
    “state”: <string>,
    “primary”: <boolean>
  }
]

For “indices_stats”:

“shards”: {
  “total”: <number>,
  “unassigned”: <number>
}

For reference, here is where we are showing shard data in the UI:

Shard legend on the index detail page

Index listing page

Shard legend on the node detail page

Node listing page

Shard totals on index detail page

Shard totals on node detail page

[elasticmachine]

Pinging @elastic/stack-monitoring (Stack monitoring)

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[chrisronline]

Quick update here.

In order to support this effort, we have two options from an Elasticsearch perspective:

1. The Metricbeat module will need to call multiple endpoints and assemble the shard data together, as the Elasticsearch team will not make changes to the existing APIs. This is less ideal, but might be the only option moving forward.
2. The Elasticsearch team exposes a new set of monitoring-specific APIs that return necessary data in the required format directly, meaning the Metricbeat module will only need to call a single API. This is the idea path but requires resources from the Elasticsearch team which may not happen.

Moving forward, we are in a holding pattern until hearing back from the Elasticsearch team on the feasibility of adding these new monitoring-specific Monitoring endpoints in 8.0.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.