Use Explain in AllocationDecider's Decisions

[s1monw]

Currently the Decision class already supports an explain parameter. It would be helpful for development, debugging and for certain error messages like in MoveAllocationCommand to log the actual reasoning behind the decision.

[dakrone]

We should add an explain parameter so people can get detailed feedback about why a shard can or cannot be allocated to a node. Something like this:

For these commands:

curl -XPOST 'localhost:9200/_cluster/reroute?explain&pretty' -d '{
  "commands" : [
    {
      "cancel" : {
        "index" : "decide", "shard" : 0, "node": "IvpoKRdtRiGrQ_WKtt4_4w"
      }
    },
    {
      "move" : {
        "index" : "decide", "shard" : 0,
        "from_node" : "IvpoKRdtRiGrQ_WKtt4_4w", "to_node" : "IvpoKRdtRiGrQ_WKtt4_4w"
      }
    }
  ]
}'

This result:

{
  "explanations" : [ {
    "command" : "cancel",
    "parameters" : {
      "index" : "decide",
      "shard" : 0,
      "node" : "IvpoKRdtRiGrQ_WKtt4_4w",
      "allow_primary" : false
    },
    "decisions" : [ {
      "decider" : "CancelAllocationCommand",
      "decision" : "NO",
      "explanation" : "can't cancel [decide][0] on node [Wysper][IvpoKRdtRiGrQ_WKtt4_4w][Xanadu.local][inet[/172.16.1.8:9300]], shard is primary and started"
    } ]
  }, {
    "command" : "move",
    "parameters" : {
      "index" : "decide",
      "shard" : 0,
      "from_node" : "IvpoKRdtRiGrQ_WKtt4_4w",
      "to_node" : "IvpoKRdtRiGrQ_WKtt4_4w"
    },
    "decisions" : [ {
      "decider" : "SameShard",
      "decision" : "NO",
      "explanation" : "shard cannot be allocated on same node [IvpoKRdtRiGrQ_WKtt4_4w] it already exists on"
    }, {
      "decider" : "Filter",
      "decision" : "YES",
      "explanation" : "node passes include/exclude/require filters"
    }, {
      "decider" : "ReplicaAfterPrimaryActive",
      "decision" : "YES",
      "explanation" : "shard is primary"
    }, {
      "decider" : "Throttling",
      "decision" : "YES",
      "explanation" : "below shard recovery limit of [2]"
    }, {
      "decider" : "Enable",
      "decision" : "YES",
      "explanation" : "allocation disabling is ignored"
    }, {
      "decider" : "Disable",
      "decision" : "YES",
      "explanation" : "allocation disabling is ignored"
    }, {
      "decider" : "Awareness",
      "decision" : "YES",
      "explanation" : "no allocation awareness enabled"
    }, {
      "decider" : "ShardsLimit",
      "decision" : "YES",
      "explanation" : "total shard limit disabled: [-1] <= 0"
    }, {
      "decider" : "NodeVersion",
      "decision" : "YES",
      "explanation" : "target node version [2.0.0-SNAPSHOT] is same or newer than source node version [2.0.0-SNAPSHOT]"
    }, {
      "decider" : "DiskThreshold",
      "decision" : "YES",
      "explanation" : "disk threshold decider disabled"
    }, {
      "decider" : "SnapshotInProgress",
      "decision" : "YES",
      "explanation" : "no snapshots are currently running"
    } ]
  } ]
}

[dakrone]

Related: #4380