Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prepares allocator decision objects for use with the allocation explain API #21691

Merged
merged 53 commits into from Dec 7, 2016
Merged
Show file tree
Hide file tree
Changes from 49 commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
6d9ef1b
This commit enhances the allocator decision result objects to enable
Nov 21, 2016
f3cc960
Improve explanations and move RebalanceDecision explanation calculation
Nov 22, 2016
d0fab3b
Serialization shouldn't care about cached decisions
Nov 22, 2016
937e9a0
remove innerWriteTo in favor of subclass calling super.writeTo(out)
Nov 22, 2016
e2c97cc
Adds assigned node name in addition to assigned node id
Nov 22, 2016
c3bca06
Changes "final" decision label to something better
Nov 22, 2016
38d1e30
fix compiler issues
Nov 22, 2016
b5c7cbc
Adds store exception to store status
Nov 22, 2016
a330c28
Removes weight info from NodeRebalanceResult and uses weight ranking
Nov 22, 2016
a78ada4
explain API won't initiate a fetch in ReplicaShardAllocator if fetch
Nov 23, 2016
d304d37
improve move decision explanation
Nov 23, 2016
313f993
changed getFinalExplanation to getExplantion()
Nov 23, 2016
7fd10ae
Use readOptionalWriteable and make Type implement Writeable
Nov 23, 2016
58e1cc2
Decisions store DiscoveryNode instead of just node id and name
Nov 24, 2016
975f064
more improvements to decision serialization
Nov 24, 2016
5bd6baf
better move decision explanation
Nov 24, 2016
a6f76b0
remove "final" from decision x-content
Nov 24, 2016
10ebbff
fix move throttled explanation
Nov 24, 2016
9b80de7
removing the lol'ness
Nov 24, 2016
a46c55a
shard store uses read/write writeable
Nov 24, 2016
690f962
Adds total delay for delayed allocation and StoreStatus AVAILABLE cha…
Nov 25, 2016
bafaeb0
improves RebalanceDecision explanations
Nov 25, 2016
64d81d1
Add SHARD_LOCK store status
Nov 25, 2016
03ba86c
fancy streams
Nov 25, 2016
90f84e7
adds hasInitiatedFetching() method to ReplicaShardsAllocator
Nov 25, 2016
9e7a734
fix more tests
Nov 25, 2016
a14e904
Separates StoreStatus from StoreReadability
Nov 25, 2016
68b7977
addresses code review comments
Nov 28, 2016
d28f7f1
Addresses code review comments
Nov 30, 2016
21296df
better sorting
Dec 1, 2016
c86477a
allows readMap to take a map supplier, with serialization tests
Dec 1, 2016
e39d96c
AbstractAllocateDecision and much code reduction as a result
Dec 1, 2016
9ddc3b8
Adds tests for Type order
Dec 1, 2016
8499f35
rebalance node explanation just provides relative weight ranking
Dec 2, 2016
9fa38be
adds node rebalance order tests and fixes a bug
Dec 2, 2016
2e54bbf
we deserve a better ranking algorithm
Dec 2, 2016
9fcf4d9
Reorder toXContent method params
Dec 2, 2016
9948fc5
move x-content generation to each decision and include current assigned
Dec 2, 2016
82fc2f9
AbstractAllocationDecision members are now protected
Dec 2, 2016
c7fc15a
assigned_node -> target_node
Dec 2, 2016
7294058
Decision constructor takes list of node decisions instead of map
Dec 2, 2016
aa02030
fix rebasing issue
Dec 3, 2016
8f390c2
(1) Merges RebalanceDecision into MoveDecision - now we have just one
Dec 3, 2016
4a0ed53
improvements to the JSON output
Dec 5, 2016
a579b19
improves decider explanations
Dec 5, 2016
021fd81
remove unneeded readMap
Dec 5, 2016
e306175
atleast -> atLeast
Dec 5, 2016
ee0ae6e
remove currentNode from MoveDecision and improve rebalance not allowed
Dec 5, 2016
37cf32c
addresses code review comments
Dec 5, 2016
e86080a
fix spacing
Dec 5, 2016
1efae15
Use an AllocationDecision enum to convey decisions to provide
Dec 6, 2016
81bbceb
else if in AllocateUnassignedDecision
Dec 7, 2016
bd7cd33
can_remain, can_move -> "yes|no"
Dec 7, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Expand Up @@ -73,7 +73,7 @@ public NodeExplanation(StreamInput in) throws IOException {
@Override
public void writeTo(StreamOutput out) throws IOException {
node.writeTo(out);
Decision.writeTo(nodeDecision, out);
nodeDecision.writeTo(out);
out.writeFloat(nodeWeight);
if (storeStatus == null) {
out.writeBoolean(false);
Expand Down Expand Up @@ -108,7 +108,9 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws
builder.field("final_decision", finalDecision.toString());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where is this "final_decision" coming from? I thought we got rid of those...

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the old explain API code. Once we plug the new explain API into the rest layer, this whole class will disappear.

builder.field("final_explanation", finalExplanation);
builder.field("weight", nodeWeight);
builder.startArray("decisions");
nodeDecision.toXContent(builder, params);
builder.endArray();
}
builder.endObject(); // end node <uuid>
return builder;
Expand Down
@@ -0,0 +1,187 @@
/*
* Licensed to Elasticsearch under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package org.elasticsearch.cluster.routing.allocation;

import org.elasticsearch.cluster.node.DiscoveryNode;
import org.elasticsearch.cluster.routing.allocation.decider.Decision;
import org.elasticsearch.cluster.routing.allocation.decider.Decision.Type;
import org.elasticsearch.common.Nullable;
import org.elasticsearch.common.io.stream.StreamInput;
import org.elasticsearch.common.io.stream.StreamOutput;
import org.elasticsearch.common.io.stream.Writeable;
import org.elasticsearch.common.xcontent.ToXContent;
import org.elasticsearch.common.xcontent.XContentBuilder;

import java.io.IOException;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;

/**
* An abstract class for representing various types of allocation decisions.
*/
public abstract class AbstractAllocationDecision implements ToXContent, Writeable {

@Nullable
protected final Type decision;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see this variable being called "decision", "finalDecision", "decisionType" in different places. Let's make this a bit more uniform. I also wonder if this is the right class type to expose as overall "result". It can just have the values "YES/NO/THROTTLE" which feel limited w.r.t AllocationStatus.

@Nullable
protected final DiscoveryNode targetNode;
@Nullable
protected final List<NodeAllocationResult> nodeDecisions;

protected AbstractAllocationDecision(@Nullable Type decision, @Nullable DiscoveryNode targetNode,
@Nullable List<NodeAllocationResult> nodeDecisions) {
this.decision = decision;
this.targetNode = targetNode;
this.nodeDecisions = nodeDecisions != null ? sortNodeDecisions(nodeDecisions) : null;
}

protected AbstractAllocationDecision(StreamInput in) throws IOException {
decision = in.readOptionalWriteable(Type::readFrom);
targetNode = in.readOptionalWriteable(DiscoveryNode::new);
nodeDecisions = in.readBoolean() ? Collections.unmodifiableList(in.readList(NodeAllocationResult::new)) : null;
}

/**
* Returns <code>true</code> if a decision was taken by the allocator, {@code false} otherwise.
* If no decision was taken, then the rest of the fields in this object cannot be accessed and will
* throw an {@code IllegalStateException}.
*/
public boolean isDecisionTaken() {
return decision != null;
}

/**
* Returns the decision made by the allocator on whether to assign the shard. If {@link #isDecisionTaken()}
* returns {@code false}, then invoking this method will throw an {@code IllegalStateException}.
*/
public Type getDecisionType() {
checkDecisionState();
return decision;
}

/**
* Get the node that the allocator will assign the shard to, unless {@link #getDecisionType()} returns
* a value other than {@link Decision.Type#YES}, in which case this returns {@code null}. If
* {@link #isDecisionTaken()} returns {@code false}, then invoking this method will throw an {@code IllegalStateException}.
*/
@Nullable
public DiscoveryNode getTargetNode() {
checkDecisionState();
return targetNode;
}

/**
* Gets the sorted list of individual node-level decisions that went into making the final decision as
* represented by {@link #getDecisionType()}. If {@link #isDecisionTaken()} returns {@code false}, then
* invoking this method will throw an {@code IllegalStateException}.
*/
@Nullable
public List<NodeAllocationResult> getNodeDecisions() {
checkDecisionState();
return nodeDecisions;
}

/**
* Gets the explanation for the decision. If {@link #isDecisionTaken()} returns {@code false}, then invoking
* this method will throw an {@code IllegalStateException}.
*/
public abstract String getExplanation();

@Override
public void writeTo(StreamOutput out) throws IOException {
out.writeOptionalWriteable(decision);
out.writeOptionalWriteable(targetNode);
if (nodeDecisions != null) {
out.writeBoolean(true);
out.writeList(nodeDecisions);
} else {
out.writeBoolean(false);
}
}

protected void checkDecisionState() {
if (isDecisionTaken() == false) {
throw new IllegalStateException("decision was not taken, individual object fields cannot be accessed");
}
}

/**
* Generates X-Content for a {@link DiscoveryNode} that leaves off some of the non-critical fields.
*/
public static XContentBuilder discoveryNodeToXContent(DiscoveryNode node, boolean outerObjectWritten, XContentBuilder builder)
throws IOException {

builder.field(outerObjectWritten ? "id" : "node_id", node.getId());
builder.field(outerObjectWritten ? "name" : "node_name", node.getName());
builder.field("transport_address", node.getAddress().toString());
if (node.getAttributes().isEmpty() == false) {
builder.startObject(outerObjectWritten ? "attributes" : "node_attributes");
for (Map.Entry<String, String> entry : node.getAttributes().entrySet()) {
builder.field(entry.getKey(), entry.getValue());
}
builder.endObject();
}
return builder;
}

/**
* Sorts a list of node level decisions by the decision type, then by weight ranking, and finally by node id.
*/
public List<NodeAllocationResult> sortNodeDecisions(List<NodeAllocationResult> nodeDecisions) {
return Collections.unmodifiableList(nodeDecisions.stream().sorted().collect(Collectors.toList()));
}

/**
* Generates X-Content for the node-level decisions, creating the outer "node_decisions" object
* in which they are serialized.
*/
public XContentBuilder nodeDecisionsToXContent(List<NodeAllocationResult> nodeDecisions, XContentBuilder builder, Params params)
throws IOException {

if (nodeDecisions != null) {
builder.startArray("node_decisions");
{
for (NodeAllocationResult explanation : nodeDecisions) {
explanation.toXContent(builder, params);
}
}
builder.endArray();
}
return builder;
}

/**
* Returns {@code true} if there is at least one node that returned a {@link Type#YES} decision for allocating this shard.
*/
protected boolean atLeastOneNodeWithYesDecision() {
if (nodeDecisions == null) {
return false;
}
for (NodeAllocationResult result : nodeDecisions) {
if (result.getNodeDecisionType() == Type.YES) {
return true;
}
}
return false;
}

}