Polarize actions based on impact vectors #332

ktkrg · 2020-08-03T18:33:11Z

Issue #:
Fixes #331

Description of changes:
This change adds the ability for the collator to group decider actions by nodes and polarize the actions based on their impact vectors.

The collator looks at various actions suggested by the deciders and first groups them by the node those actions impact.

For each node, if there is an action suggested that would go in the direction of decreasing pressure, then only choose(a la polarize) pressure decreasing actions. Only if there is no pressure decreasing action suggested for a node, then choose actions that increase pressure.

Tests:
Unit tests and tested on docker.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

yojs · 2020-08-06T18:36:44Z

...n/opendistro/elasticsearch/performanceanalyzer/decisionmaker/deciders/collator/Collator.java

+ * <p>The collator prunes them to ensure we only take actions that either increase, or decrease
+ * pressure on a particular node. To resolve conflicts, we prefer stability over performance.
+ */
+public class Collator extends Decider {


There can only be one collator node in a graph right ? Can we add that to the comments ?

yojs · 2020-08-06T18:37:20Z

...n/opendistro/elasticsearch/performanceanalyzer/decisionmaker/deciders/collator/Collator.java

+    return proposedActions;
+  }
+
+  private List<Action> polarize(final NodeKey nodeKey, List<Action> actions) {


maybe we can make actions final as well ?

done. Took care of the other parameters as well.

yojs · 2020-08-06T18:40:05Z

...n/opendistro/elasticsearch/performanceanalyzer/decisionmaker/deciders/collator/Collator.java

+      if (impactVector.getImpact()
+                      .values()
+                      .stream()
+                      .anyMatch(impact -> impact == Impact.INCREASES_PRESSURE)) {


Functional programming !! yay !

yojs · 2020-08-06T18:57:07Z

...earch/performanceanalyzer/decisionmaker/deciders/collator/SingleNodeImpactActionGrouper.java

+                                                                    .get(0), k -> new ArrayList<>())
+                                             .add(action));
+    return actionsByNodeId;


Do we need to turn on some checkstyle rukes, this should have caused an error ?

the .get(0) is aligned with .impactedNodes() because they're chained calls, and similarly, .add(action) is aligned with .computeIfAbsent() because they're chained calls.

This is what the formatter did and I thought it was helpful 😅 I'm using the GoogleStyle scheme. Let me know if I should use another formatter.

yojs · 2020-08-06T18:58:01Z

...h/performanceanalyzer/decisionmaker/deciders/collator/SingleNodeImpactActionGrouperTest.java

+    assertEquals(1, groupedActions.get(nodeKey1).size());
+    assertEquals(singleNodeImpactAction1, groupedActions.get(nodeKey1).get(0));
+  }
+}


A new line in the end ?

yojs · 2020-08-06T19:00:39Z

Can you add a label for the PR ?

yojs · 2020-08-06T19:02:38Z

In the PR description

first groups them by the node

Unfortunately, node is an overloaded term which can mean a graph node or a cluster node. Can we clarify ?

khushbr · 2020-08-06T18:44:42Z

...ndistro/elasticsearch/performanceanalyzer/decisionmaker/deciders/collator/ActionGrouper.java

+public interface ActionGrouper {
+
+  /**
+   * Groups the given list of actions by the nodes they impact..


nit: Extra full stop at the end.

khushbr · 2020-08-06T18:54:22Z

...n/opendistro/elasticsearch/performanceanalyzer/decisionmaker/deciders/collator/Collator.java

+      for (final Decider decider : deciders) {
+        List<Decision> decisions = decider.getFlowUnits();
+        if (decisions != null) {
+          decisions.forEach(decision -> proposedActions.addAll(decision.getActions()));


Can decision.getActions() return null or empty value ? Do we want to filter for them?

not sure if any actions return null values, but added a null check any way.

khushbr · 2020-08-06T18:57:40Z

...n/opendistro/elasticsearch/performanceanalyzer/decisionmaker/deciders/collator/Collator.java

+    List<Action> proposedActions = getAllProposedActions();
+    Map<NodeKey, List<Action>> actionsByNode = actionGrouper.groupByNodeId(proposedActions);
+    List<Action> prunedActions = new ArrayList<>();
+    actionsByNode.forEach((k, v) -> prunedActions.addAll(polarize(k, v)));


can we rename the variables k, v here to be more expressive of the purpose.

khushbr · 2020-08-06T19:00:07Z

...n/opendistro/elasticsearch/performanceanalyzer/decisionmaker/deciders/collator/Collator.java

+                      .anyMatch(impact -> impact == Impact.INCREASES_PRESSURE)) {
+        pressureIncreasingActions.add(action);
+      } else {
+        pressureNonIncreasingActions.add(action);


given "If there are any actions that decrease pressure for a node, prefer that over list of actions that increase pressure.", should we break here when we find a pressureNonIncreasingActions ?

there might be other pressure decreasing actions beyond the one that increases it, we need to add that as well, so we can't break early.

khushbr · 2020-08-06T19:03:57Z

...earch/performanceanalyzer/decisionmaker/deciders/collator/SingleNodeImpactActionGrouper.java

+  @NonNull public Map<NodeKey, List<Action>> groupByNodeId(@NonNull List<Action> actions) {
+    final Map<NodeKey, List<Action>> actionsByNodeId = new HashMap<>();
+    actions.stream()
+           .filter(action -> action.impactedNodes().size() == 1)


so, each action can be associated with 1 or 0 impacted Nodes ? Just wanted to clarify this.

Yeah, is that not the understanding today? This class only considers those actions that impact exactly one node. The multi node impact action grouping will be done in the next PR.

@yojs brought up a similar point, do we create a single action with multiple nodes in the impacted nodes when we want to say decrease cache size for node1, node2, node3 or do we create multiple actions for each node we want to decrease the cache size for?

I clarified this with others and we will have actions which impact a single node and then another set of action which are for multiple nodes.

Taking the above example, cache increase action will be per node but for cache decrease, we will have 1 action with multiple nodes in the impactedNodes.

Actions can impact a single node or multiple nodes - this depends on the type of action. For a given action type (a class implementing Action interface), this stays fixed.

@khushbr's example above is slightly incorrect. ModifyCacheAction impacts only a single node. If we want to increase or decrease caches across multiple nodes, the decider will have to create multiple such actions.

There will be other types of actions in future, like MoveShard, which can impact multiple nodes (src node and dest node) or SplitShard which impacts even more nodes. For those actions, impactedNodes will contain all the nodes impacted and impact() will return the impactVector for each node - e.g. for MoveShard impact() will indicate that pressure gets reduced on src node and increased on the destination node.

yojs · 2020-08-06T19:04:18Z

...ndistro/elasticsearch/performanceanalyzer/decisionmaker/deciders/collator/ActionGrouper.java

+   * @param actions The list of actions that need to be grouped.
+   * @return A map of actions grouped by nodes they impact.
+   */
+  @NonNull Map<NodeKey, List<Action>> groupByNodeId(@NonNull final List<Action> actions);


Can we rename the method name to include InstanceName ?

yup, renamed it to groupByInstanceId.

yojs · 2020-08-06T19:08:28Z

...n/opendistro/elasticsearch/performanceanalyzer/decisionmaker/deciders/collator/Collator.java

+  private final ActionGrouper actionGrouper;
+
+  public Collator(long evalIntervalSeconds, Decider... deciders) {
+    this(evalIntervalSeconds, new SingleNodeImpactActionGrouper(), deciders);


We plan to remove the evalIntervalSeconds as we wanted the scheduler to be simple - it would call each graphNode at every tick and then the node should decide how to handle if this is before the periodicity. In that vein, can we add a constant in Node class and use that and not even take evalIntervalSeconds as a constructor parameter ?

Removed evalIntervalSeconds ctor parameter in Collator. It should be one less graph node to worry about when we remove the parameter permanently :)

yojs · 2020-08-06T19:11:38Z

...n/opendistro/elasticsearch/performanceanalyzer/decisionmaker/deciders/collator/Collator.java

+  }
+
+  @Override
+  public Decision operate() {


I think it might be a good idea to add a javadoc comment for the operate() as this is the center piece of the RCAGraph node ?

yojs · 2020-08-06T19:17:38Z

...earch/performanceanalyzer/decisionmaker/deciders/collator/SingleNodeImpactActionGrouper.java

+public class SingleNodeImpactActionGrouper implements ActionGrouper {
+
+  @Override
+  @NonNull public Map<NodeKey, List<Action>> groupByNodeId(@NonNull List<Action> actions) {


So, we want the collator to come up with one action per node or one action per resource per node ?

Collator does not filter beyond polarizing the actions. It will return as many as possible after making sure that they align in terms of the impact on the instance so that the publisher and other components can filter based on other parameters.

yojs · 2020-08-06T19:33:52Z

...earch/performanceanalyzer/decisionmaker/deciders/collator/SingleNodeImpactActionGrouper.java

+  @NonNull public Map<NodeKey, List<Action>> groupByNodeId(@NonNull List<Action> actions) {
+    final Map<NodeKey, List<Action>> actionsByNodeId = new HashMap<>();
+    actions.stream()
+           .filter(action -> action.impactedNodes().size() == 1)


Can we add a comment why are we filtering out the actions that ask for the same action to be performed on multiple nodes ? Is that rationale that we do it only one node per iteration ?

Interesting point, don't we create multiple actions in such cases?

Looks like we are creating one action per node even if its the same action for multiple nodes, added a comment.

codecov · 2020-08-06T20:58:03Z

Codecov Report

Merging #332 into master will increase coverage by 0.19%.
The diff coverage is 90.62%.

@@             Coverage Diff              @@
##             master     #332      +/-   ##
============================================
+ Coverage     69.92%   70.11%   +0.19%     
- Complexity     2242     2282      +40     
============================================
  Files           301      303       +2     
  Lines         13481    13605     +124     
  Branches       1116     1133      +17     
============================================
+ Hits           9426     9539     +113     
- Misses         3690     3691       +1     
- Partials        365      375      +10

Impacted Files	Coverage Δ	Complexity Δ
...anceanalyzer/decisionmaker/deciders/Publisher.java	`83.78% <ø> (ø)`	`12.00 <0.00> (ø)`
...h/performanceanalyzer/rca/framework/core/Node.java	`88.13% <ø> (ø)`	`29.00 <0.00> (ø)`
...isionmaker/deciders/collator/ImpactAssessment.java	`87.50% <87.50%> (ø)`	`18.00 <18.00> (?)`
...yzer/decisionmaker/deciders/collator/Collator.java	`90.38% <90.38%> (ø)`	`10.00 <10.00> (?)`
...ecisionmaker/deciders/collator/ImpactAssessor.java	`96.29% <96.29%> (ø)`	`12.00 <12.00> (?)`
...analyzer/rca/store/ElasticSearchAnalysisGraph.java	`100.00% <100.00%> (ø)`	`7.00 <0.00> (ø)`
.../performanceanalyzer/rca/framework/api/Metric.java	`76.19% <0.00%> (-4.77%)`	`8.00% <0.00%> (-1.00%)`
...csearch/performanceanalyzer/rca/RcaController.java	`83.00% <0.00%> (-0.98%)`	`46.00% <0.00%> (ø%)`
...rch/performanceanalyzer/config/SettingsHelper.java	`20.00% <0.00%> (ø)`	`1.00% <0.00%> (ø%)`
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fb18d66...a4fbb59. Read the comment docs.

vigyasharma · 2020-08-13T21:42:18Z

...n/opendistro/elasticsearch/performanceanalyzer/decisionmaker/deciders/collator/Collator.java

+        List<Decision> decisions = decider.getFlowUnits();
+        if (decisions != null) {
+          decisions.forEach(decision -> {
+            if (decision.getActions() != null) {


We should use decision.isEmpty() here

will change in the next rev

vigyasharma · 2020-08-13T21:48:49Z

...earch/performanceanalyzer/decisionmaker/deciders/collator/SingleNodeImpactActionGrouper.java

+  @NonNull public Map<NodeKey, List<Action>> groupByNodeId(@NonNull List<Action> actions) {
+    final Map<NodeKey, List<Action>> actionsByNodeId = new HashMap<>();
+    actions.stream()
+           .filter(action -> action.impactedNodes().size() == 1)


Actions can impact a single node or multiple nodes - this depends on the type of action. For a given action type (a class implementing Action interface), this stays fixed.

@khushbr's example above is slightly incorrect. ModifyCacheAction impacts only a single node. If we want to increase or decrease caches across multiple nodes, the decider will have to create multiple such actions.

There will be other types of actions in future, like MoveShard, which can impact multiple nodes (src node and dest node) or SplitShard which impacts even more nodes. For those actions, impactedNodes will contain all the nodes impacted and impact() will return the impactVector for each node - e.g. for MoveShard impact() will indicate that pressure gets reduced on src node and increased on the destination node.

vigyasharma · 2020-08-13T22:01:27Z

...n/opendistro/elasticsearch/performanceanalyzer/decisionmaker/deciders/collator/Collator.java

+      // Classify the action as pressure increasing action if the impact for any dimension is
+      // increasing pressure.
+      if (impactVector.getImpact()


This classifies an increase across any dimension as increasing pressure. Shouldn't we check on a per dimension basis? Otherwise there is no value in having different dimensions in the impact vector.

Suppose you have an action to offload some data from heap to a file on disk. This would reduce heap but increase disk pressure. If there is no other action trying to decrease disk pressure, then the overall disk on the node is not under contention and we should let this action go through.

Makes sense, I assumed for the first iteration we would not have such actions and had not thought of polarizing at a dimension level. I will change it.

vigyasharma · 2020-08-13T22:44:04Z

...n/opendistro/elasticsearch/performanceanalyzer/decisionmaker/deciders/collator/Collator.java

+    return proposedActions;
+  }
+
+  private List<Action> polarize(final NodeKey nodeKey, final List<Action> actions) {


It seems to me that this will need non-trivial changes when we have actions impacting multiple nodes. If an action is purged because of pressure conflict on one of the nodes, it must be removed from the list of all of the nodes.

Should polarize() then move to ActionGrouper? Is the overall above structure right for future use cases?

My ideas for solving this were more around compiling all impact vectors into a matrix and doing a second pass through it to accept or prune actions. Something on these lines:

Suppose the collator received 2 actions, moveShard from node A to node B, and moveShard from node B to node C

actions = [MoveShard: A -> B, MoveShard: B -> C] Impact vectors: actions[0] :: MoveShard: A -> B impactedNodes = [A, B] A: [CPU:dec, Heap: dec, RAM: dec, NW: dec, Disk: dec] B: [CPU:inc, Heap: inc, RAM: inc, NW: inc, Disk: inc] actions[1] :: MoveShard: B -> C impactedNodes = [B, C] B: [CPU:dec, Heap: dec, RAM: dec, NW: dec, Disk: dec] C: [CPU:inc, Heap: inc, RAM: inc, NW: inc, Disk: inc]

First we create an impact matrix for the cluster:

node A has only decrease across each dim

B has 1 inc and 1 dec action,

C has 1 inc action for each dim

A B C cpu i=0,d=1 i=1,d=1 i=1,d=0 heap i=0,d=1 i=1,d=1 i=1,d=0 ram i=0,d=1 i=1,d=1 i=1,d=0 nw i=0,d=1 i=1,d=1 i=1,d=0 disk i=0,d=1 i=1,d=1 i=1,d=0

Now we run the actions through this matrix again.

If an action is increasing pressure on a dim on a node, it is only allowed
if there is no other action trying to dec that dim on the node. i.e. d=0 for that node,dim.

When an action gets purged in this second pass, we remove it from the matrix by reducing the node,dim counters.

In above e.g. we process MoveShard(A->B) -

It cannot go through because d=1 for some dims on node B

We purge MoveShard(A->B)

Counters on A and B are updated. All counters in A become i=0, d=0. B becomes i=0,d=1

Now we process MoveShard(B->C) -

It increases pressure on C but there is no one decreasing pressure (d=0 for all dims on C), so it can go through

On B it is dec pressure anyway so it has a clear pass.

Hence MoveShard(B->C) gets picked.

This will also resolve cycles like A->B, B->C, C->A by picking either one or two of the actions (depending on order of 2nd pass).

It is okay if we handle multi-node actions in a separate PR later. You may also use different/better algorithms for polarizing impact vectors. Just wanted to check if the current Collator::polarize() and ActionGrouper::groupByInstanceId() is the right approach for future use cases, like the moveShard, splitShard etc..

It will also be good to at least handle the dimension level comparisons right away, as they help establish a structure for all use cases.

Thanks for this sage advice, I had an unnecessarily complex system and I was able to rip it out until I got to something similar to what you've proposed here :) I've updated the PR to reflect these changes.

vigyasharma

Looks good @ktkrg, clean and well documented. Thanks for making these changes.

vigyasharma · 2020-08-21T06:40:22Z

...n/opendistro/elasticsearch/performanceanalyzer/decisionmaker/deciders/collator/Collator.java

+   * to most.
+   */
+  @VisibleForTesting
+  static final class ImpactBasedActionComparator implements Comparator<Action>, Serializable {


Interesting implementation.

vigyasharma · 2020-08-21T06:51:40Z

...stro/elasticsearch/performanceanalyzer/decisionmaker/deciders/collator/ImpactAssessment.java

+      @NonNull final ImpactVector impactVector) {
+    boolean isAligned = true;
+
+    // If this is an action that increases pressure along some dimension for this node, and the


This is a useful comment, thanks for adding

vigyasharma · 2020-08-21T06:52:06Z

...stro/elasticsearch/performanceanalyzer/decisionmaker/deciders/collator/ImpactAssessment.java

+   */
+  public boolean checkAlignmentAcrossDimensions(@NonNull final String actionName,
+      @NonNull final ImpactVector impactVector) {
+    boolean isAligned = true;


nit: move to after the comment string?

vigyasharma · 2020-08-21T06:54:52Z

...stro/elasticsearch/performanceanalyzer/decisionmaker/deciders/collator/ImpactAssessment.java

+    final Map<Dimension, Impact> impactMap = impactVector.getImpact();
+    for (final Map.Entry<Dimension, Impact> entry : impactMap.entrySet()) {
+      final Impact impactOnDimension = entry.getValue();


can we use impactMap.foreach(dimension, impact) {...} ?

We can't use forEach here because we will be updating a local variable inside the body of the lambda and it needs the isAligned value to be effectively final. Since local variables are not effectively final, we'll need to declare an array or a list of size=1 to hold the isAligned value.

I chose the verbose way to avoid some confusion there.

vigyasharma · 2020-08-21T06:56:45Z

...stro/elasticsearch/performanceanalyzer/decisionmaker/deciders/collator/ImpactAssessment.java

+        List<String> pressureDecreasingActions = perDimensionPressureDecreasingActions
+            .getOrDefault(entry.getKey(), Collections.emptyList());
+        isAligned = pressureDecreasingActions.isEmpty();


How about:

isAligned = !pressureDecreasingActions.contains(dimension);

vigyasharma · 2020-08-21T06:57:19Z

...stro/elasticsearch/performanceanalyzer/decisionmaker/deciders/collator/ImpactAssessment.java

+    map.computeIfAbsent(dimension,
+        dim -> new ArrayList<>()).add(actionName);


nit: single line?

vigyasharma · 2020-08-21T06:59:28Z

...endistro/elasticsearch/performanceanalyzer/decisionmaker/deciders/collator/CollatorTest.java

+
+    return impactVector;
+  }
+}


nit: add newline

khushbr

LGTM!

yojs · 2020-08-24T18:22:32Z

Do you think we can add integ tests for this ?

ktkrg added 5 commits August 3, 2020 11:31

Polarize actions based on impact vectors

8ff2566

Merge from master

5e58403

Merge from master

7649d14

Fix import order

95f958d

Use InstanceDetails.Id and Ip instead of String

490d7f2

ktkrg requested review from vigyasharma, khushbr and yojs August 6, 2020 04:03

yojs reviewed Aug 6, 2020

View reviewed changes

khushbr reviewed Aug 6, 2020

View reviewed changes

yojs reviewed Aug 6, 2020

View reviewed changes

ktkrg added the enhancement Enhancements to existing codebase label Aug 6, 2020

Address PR comments

3dcd77c

Refactor collator constructor to not take evalInterval

19d6f2c

ktkrg requested review from yojs and khushbr August 6, 2020 21:38

khushbr approved these changes Aug 6, 2020

View reviewed changes

vigyasharma suggested changes Aug 13, 2020

View reviewed changes

ktkrg added 4 commits August 17, 2020 14:24

Merge from master

9ce128c

Handle multi node impacting actions in the collator

002179c

Merge from master

7d9484a

Add missing javadoc

8919220

ktkrg requested review from vigyasharma and khushbr August 20, 2020 18:52

vigyasharma previously approved these changes Aug 21, 2020

View reviewed changes

khushbr previously approved these changes Aug 24, 2020

View reviewed changes

Address review comments

a4fbb59

ktkrg dismissed stale reviews from khushbr and vigyasharma via a4fbb59 August 24, 2020 22:24

ktkrg self-assigned this Aug 24, 2020

ktkrg requested review from vigyasharma and khushbr August 28, 2020 18:25

vigyasharma approved these changes Aug 28, 2020

View reviewed changes

khushbr approved these changes Aug 31, 2020

View reviewed changes

ktkrg merged commit 8a806d2 into master Aug 31, 2020

		map.computeIfAbsent(dimension,
		dim -> new ArrayList<>()).add(actionName);

Polarize actions based on impact vectors #332

Polarize actions based on impact vectors #332

Conversation

ktkrg commented Aug 3, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yojs commented Aug 6, 2020

yojs commented Aug 6, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

khushbr Aug 6, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ktkrg Aug 6, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Aug 6, 2020 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vigyasharma Aug 13, 2020 • edited Loading

Choose a reason for hiding this comment

Suppose the collator received 2 actions, moveShard from node A to node B, and moveShard from node B to node C

First we create an impact matrix for the cluster:

Now we run the actions through this matrix again.

When an action gets purged in this second pass, we remove it from the matrix by reducing the node,dim counters.

Choose a reason for hiding this comment

vigyasharma left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

khushbr left a comment

Choose a reason for hiding this comment

yojs commented Aug 24, 2020

ktkrg commented Aug 3, 2020 •

edited

Loading

khushbr Aug 6, 2020 •

edited

Loading

ktkrg Aug 6, 2020 •

edited

Loading

codecov bot commented Aug 6, 2020 •

edited

Loading

vigyasharma Aug 13, 2020 •

edited

Loading