Better task balancing #1482

matush-v · 2017-03-30T19:07:16Z

🚧 This is a WIP for task balancing.

The general idea is that a pendingTask will calculate the best offer by scoring each offer and choosing the best score. Right now, the top score is 1.00 assuming the slave has nothing on it.

Score is weighted based on 2 real criteria: current resource usage for the same request type and current resource availability. I choose weights based on what I thought might be important, but it can be changed. I thought mem would be the most important so I weighted it a bit higher:

requestTypeCpuWeight = 0.20;
requestTypeMemWeight = 0.30;
freeCpuWeight = 0.20;
freeMemWeight = 0.30;

This only scores based on what's running on the slave. It does not look at the acceptedPendingTasks for an offer.

@ssalinas

matush-v · 2017-03-30T19:13:19Z

...arityService/src/main/java/com/hubspot/singularity/mesos/SingularityMesosOfferScheduler.java

+          double score = score(offerHolder, stateCache, tasksPerOfferPerRequest, taskRequestHolder, getUsagesPerRequestTypePerSlave());
+          if (score > 0) {
+            // todo: can short circuit here if score is high enough
+            scorePerOffer.put(offerHolder, score);


Thought we might want to have a value that's definitely good enough to just accept instead of continue evaluating

ssalinas · 2017-03-30T20:30:45Z

I like the idea of a scoring system overall. Some comments on specific logic things I'll make later since this is still WIP. Overall comments though:

The time past due for a task should probably also factor in to the scoring. (i.e. if it was supposed to run 5 minutes ago, the definition of 'good enough' is different than if it was supposed to start 2 seconds ago)
There should probably also be a measure of how many offers we have looked at while trying to schedule this task. We may only get 1,2 etc offers at a time and not have a wholistic view of resources. So if we have looked at a number of them already, the bar of 'good enough' should start to get lower.
We should definitely be aware of computation time. The offer processing loop is already one of our slower areas. Anything we can do to pre-process this data and require less calculation at offer evaluation time will be a big help

matush-v · 2017-04-05T19:05:27Z

...arityService/src/main/java/com/hubspot/singularity/mesos/SingularityMesosOfferScheduler.java


-        taskManager.createTaskAndDeletePendingTask(zkTask);
+  private double minScore(SingularityTaskRequest taskRequest) {
+    double minScore = 0.80;


this can be adjusted as necessary. I thought an 80% match might be a good starting point, but we could def reduce it

matush-v · 2017-04-05T19:21:39Z

This has been updated as follows:

Before

All offers would be scored and of all scores above 0, the best scored offer would be accepted by the task

After

All offers are still scored, but the minimum score acceptable depends on the task's overdue milliseconds and number of offers a task has not accepted.

Currently, the overdue time and offer count have a max of 10 min and 20 attempts, respectively. The min score is based on the ratio of curOverdueTime:maxOverdueTime and curAttempts:maxAttempts

The offer attempts count is any offer that was considered, not just offers that scored too low. So an offer that didn't have enough resources to satisfy the task will still be counted.

I picked these numbers based on generalizations. We will likely have to tune these

matush-v · 2017-04-05T19:23:30Z

...arityService/src/main/java/com/hubspot/singularity/mesos/SingularitySlaveAndRackManager.java

      return SlaveMatchState.SLAVE_ATTRIBUTES_DO_NOT_MATCH;
    }

    final SlavePlacement slavePlacement = taskRequest.getRequest().getSlavePlacement().or(configuration.getDefaultSlavePlacement());

    if (!taskRequest.getRequest().isRackSensitive() && slavePlacement == SlavePlacement.GREEDY) {
+      // todo: account for this or let this behavior continue?
      return SlaveMatchState.NOT_RACK_OR_SLAVE_PARTICULAR;


I didn't know if we would need to account for any rack sensitivity outside of the existing checks done in the scheduler

matush-v · 2017-04-14T15:50:20Z

@ssalinas Got some specific logic tests in. Let me know if there's a piece I missed that should be added. I'm going to continue to try to get some full logic tests in as well

ssalinas · 2017-04-19T20:45:09Z

SingularityBase/src/main/java/com/hubspot/singularity/SingularitySlaveUsage.java

 import com.fasterxml.jackson.annotation.JsonCreator;
 import com.fasterxml.jackson.annotation.JsonProperty;

 public class SingularitySlaveUsage {

+  public static final String CPU_USED = "cpusUsed";
+  public static final String MEMORY_BYTES_USED = "memoryRssBytes";
+  public static final long BYTES_PER_MEGABYTE = 1024L * 1024L;


was about to comment that there must be some type of easy class/enum for this like there is with TimeUnit, but apparently there isn't... weird...

Yeah, I was sad to see there wasn't a lib method for this too 😢

ssalinas · 2017-04-19T20:48:08Z

SingularityBase/src/main/java/com/hubspot/singularity/SingularitySlaveUsage.java

+  }
+
+  public double getCpusUsedForRequestType(RequestType type) {
+    return usagePerRequestType.get(type).get(CPU_USED).doubleValue();


Maybe another enum is more appropriate for CPU_USED/MEMORY_BYTES_USED ?

agreed, mapping would be clearer then too 👍

ssalinas · 2017-04-19T20:48:34Z

SingularityBase/src/main/java/com/hubspot/singularity/SingularitySlaveUsage.java

+    return usagePerRequestType;
+  }
+
+  public double getCpusUsedForRequestType(RequestType type) {


this and getMemBytesUsedForRequestType are unused methods

ssalinas · 2017-04-19T20:55:53Z

SingularityService/src/main/java/com/hubspot/singularity/scheduler/SingularityUsagePoller.java

  }

  @Override
  public void runActionOnPoll() {
    final long now = System.currentTimeMillis();
+    Map<RequestType, Map<String, Number>> usagesPerRequestType = new HashMap<>();


wouldn't we want this to be per-slave, not overall?

This should be per slave. This poller loops through each slave and creates a new SingularitySlaveUsage with the stats for that slave

ssalinas · 2017-04-20T14:53:10Z

...arityService/src/main/java/com/hubspot/singularity/mesos/SingularityMesosOfferScheduler.java

+        Map<SingularityOfferHolder, Double> scorePerOffer = new HashMap<>();
+        double minScore = minScore(taskRequestHolder.getTaskRequest(), offerMatchAttemptsPerTask, System.currentTimeMillis());
+
+        LOG.info("Minimum score for task {} is {}", taskRequestHolder.getTaskRequest().getPendingTask().getPendingTaskId().getId(), minScore);


probably can be lower than info level here

ssalinas · 2017-04-20T14:54:52Z

...arityService/src/main/java/com/hubspot/singularity/mesos/SingularityMesosOfferScheduler.java

+            continue;
+          }
+
+          double score = score(offerHolder, stateCache, tasksPerOfferPerRequest, taskRequestHolder, getSlaveUsage(currentSlaveUsages, offerHolder.getOffer().getSlaveId().getValue()));


for clarity, maybe something like 'hostScore' here? The score is for the particular slave, not necessarily about the offer

I'm not sure about the naming here. We do look at the slave's utilization to score the offer, but we are still scoring the offer itself since offers aren't uniquely 1:1 for a slave (e.g. 2 offers for the same slave).

The slave utilization weight will be the same for all offers on the same slave, but the offer resources will be different per offer. So, it seems to me that we're scoring the offer in this class rather than the slave itself

ssalinas · 2017-04-20T15:01:11Z

...arityService/src/main/java/com/hubspot/singularity/mesos/SingularityMesosOfferScheduler.java

+  @VisibleForTesting
+  double minScore(SingularityTaskRequest taskRequest, Map<String, Integer> offerMatchAttemptsPerTask, long now) {
+    double minScore = 0.80;
+    int maxOfferAttempts = 20;


another that would be nice to have ocnfigurable

ssalinas · 2017-04-20T15:01:57Z

...arityService/src/main/java/com/hubspot/singularity/mesos/SingularityMesosOfferScheduler.java

-        final SingularityTask task = mesosTaskBuilder.buildTask(offerHolder.getOffer(), offerHolder.getCurrentResources(), taskRequest, taskRequestHolder.getTaskResources(), taskRequestHolder.getExecutorResources());
+  @VisibleForTesting
+  double score(Offer offer, SingularityTaskRequest taskRequest, Optional<SingularitySlaveUsageWithId> maybeSlaveUsage) {
+    double requestTypeCpuWeight = 0.20;


Let's make these configurable, maybe another object in the configuration yaml?

Yup, I was in progress on this (now committed), but I kept the fields under SingularityConfiguration since I saw a lot of other stuff in there as well (e.g. caching). We could pull it into an OfferConfiguration file if you think that'd be better for organization

ssalinas · 2017-04-20T15:02:20Z

...arityService/src/main/java/com/hubspot/singularity/mesos/SingularityMesosOfferScheduler.java

-      if (matchesResources && slaveMatchState.isMatchAllowed()) {
-        final SingularityTask task = mesosTaskBuilder.buildTask(offerHolder.getOffer(), offerHolder.getCurrentResources(), taskRequest, taskRequestHolder.getTaskResources(), taskRequestHolder.getExecutorResources());
+  @VisibleForTesting
+  double score(Offer offer, SingularityTaskRequest taskRequest, Optional<SingularitySlaveUsageWithId> maybeSlaveUsage) {


Let's go over this one in-person, think we are getting close, just easier to chat than typing a novel in github ;)

matush-v · 2017-05-23T18:46:32Z

SingularityService/src/main/java/com/hubspot/singularity/data/UsageManager.java

@@ -31,22 +32,30 @@

  private static final String SLAVE_PATH = ROOT_PATH + "/slaves";
  private static final String TASK_PATH = ROOT_PATH + "/tasks";
+  private static final String USAGE_SUMMARY_PATH = ROOT_PATH + "/summary";


@ssalinas can you take a look at this piece? Just wanna make sure I got this set up right

ssalinas · 2017-05-23T18:49:54Z

SingularityService/src/main/java/com/hubspot/singularity/data/UsageManager.java

@@ -123,6 +140,10 @@ public SingularityCreateResult saveSpecificTaskUsage(String taskId, SingularityT
    return save(getSpecificTaskUsagePath(taskId, usage.getTimestamp()), usage, taskUsageTranscoder);
  }

+  public SingularityCreateResult saveSpecificClusterUtilization(SingularityClusterUtilization utilization) {
+    return save(getSpecificClusterUtilizationPath(utilization.getTimestamp()) , utilization, clusterUtilizationTranscoder);


is there any point at which we would want a history of these? If we don't need a history of summaries, we might as well save the data to the summary path (without timestamp) and just overwrite when there is new data.

I didn't see a reason to at this point. Saw the others were saving them so it seemed okay since we only save up to 5 points. I think it's safe to just have the one point. Keeps it simpler

ssalinas · 2017-05-23T18:50:17Z

SingularityService/src/main/java/com/hubspot/singularity/data/UsageManager.java

@@ -95,6 +104,10 @@ private String getCurrentSlaveUsagePath(String slaveId) {
    return ZKPaths.makePath(getSlaveUsagePath(slaveId), CURRENT_USAGE_NODE_KEY);
  }

+  private String getSpecificClusterUtilizationPath(long timestamp) {


why SpecificClusterUtilization instead of just getUsageSummaryPath or something like that?

the "specific" keyword was the pattern the other items were using so I kept that since the timestamp specified which one to grab. I can drop the historical data and then rename it

matush-v · 2017-05-23T19:17:17Z

SingularityService/src/main/java/com/hubspot/singularity/scheduler/SingularityUsagePoller.java

+          } catch (InvalidSingularityTaskIdException e) {
+            LOG.error("Couldn't get SingularityTaskId for {}", taskUsage);
+            continue;
+          }


added this try catch for the potentially incorrect taskId types

matush-v · 2017-05-23T19:22:35Z

Before

Min score was configurable by the user and would be reduced as tasks were delayed and offer match attempts were rejected.

After

Min score is calculated based on the overall utilization of the cluster. memUsed / memTotal and cpuUsed / cpuTotal. This will help give us a min score closer to the actual offer scores. Task delays and match attempts will still reduce the min score

matush-v · 2017-05-31T20:36:33Z

...arityService/src/main/java/com/hubspot/singularity/mesos/SingularityMesosOfferScheduler.java

@@ -130,12 +131,13 @@ public SingularityMesosOfferScheduler(MesosConfiguration mesosConfiguration,

    while (!pendingTaskIdToTaskRequest.isEmpty() && addedTaskInLastLoop && canScheduleAdditionalTasks(taskCredits)) {
      addedTaskInLastLoop = false;
+      double maxTaskMillisPastDue = maxTaskMillisPastDue(SingularityScheduledTasksInfo.getInfo(taskManager.getPendingTasks(), configuration.getDeltaAfterWhichTasksAreLateMillis()).getMaxTaskLagMillis());


I think we'll need a better name for this variable. It's the max possible task lag before we decide we're going to take any offer we can get. I feel maxTaskMillisPastDue has some confusing overlap with maxTaskLag; which is the current highest lag time for pending tasks

matush-v · 2017-05-31T22:48:49Z

Before

We had a configured value for the max possible task lag before we accepted any offer that matched a task. The default was 5 minutes.

After

The max possible task lag (maxTaskMillisPastDue) is determined by a simple point-slope formula that uses the maxTaskLag and the existing deltaAfterWhichTasksAreLateMillis.
maxTaskMillisPastDue = (-180 000 / deltaAfterWhichTasksAreLateMillis) * maxTaskLag + 180 000 where 180,000 is 3 min in milliseconds

A task with no lag will result in a 3 min maxTaskMillisPastDue. Any lag will linearly reduce the maxTaskMillisPastDue. The minimum possible maxTaskMillisPastDue is 1 millisecond.

Since we reduce the minScore by how long a task is past due: higher maxTaskLag => lower maxTaskMillisPastDue => lower minScore

It's also important to note that we will actually start accepting any matching offer (minScore == 0) before the maxTaskLag reaches deltaAfterWhichTasksAreLateMillis. We can use this formula to see what the values would be for different deltas.
180 000 = l + (180 000 / d) * l where l = the task lag at which we'll accept any offer and d = deltaAfterWhichTasksAreLateMillis

ssalinas · 2017-06-08T16:48:15Z

@darcatron I think this one is good to merge. Going to get this in master so we can continue to build off of the resource usage updates without endless merge conflicts 👍 . Thanks for all the work on this one

matush-v added 7 commits March 30, 2017 12:38

get usage per request type

534b899

refactor

3b74937

score offers and choose best score

6601fb1

handling missing usage data

5292764

refactor

a4c6800

correct weight

6d663d6

correct free resource scoring

6d8a0b8

matush-v commented Mar 30, 2017

View reviewed changes

matush-v added 5 commits April 4, 2017 18:46

add usage by request type

745e71f

add usage by request type

31ec885

get usages for specific slaveIds

741a1d1

move usage by request type collection to usage poller

a855e57

consider time overdue and offers passed when picking best offer

f90bb4b

matush-v commented Apr 5, 2017

View reviewed changes

add score logging

9468d23

matush-v commented Apr 5, 2017

View reviewed changes

matush-v added 3 commits April 11, 2017 14:13

switch to bytes

8142c84

specifiy mb vs bytes

b56d6f8

add tests and fix logic errors

2a907c7

ssalinas reviewed Apr 20, 2017

View reviewed changes

matush-v force-pushed the task-juggling branch from 2f8d3d0 to e145757 Compare April 20, 2017 17:22

matush-v added 2 commits April 20, 2017 13:48

make offer scoring configurable

b6c0108

switch to enum for resource usages

d7867e9

matush-v force-pushed the task-juggling branch from e145757 to d7867e9 Compare April 20, 2017 17:50

matush-v added 3 commits April 26, 2017 23:28

track tasks that are considered long running

e5290d6

add configs for task balancing

e49d718

score based on how long a task runs

689ca42

matush-v added 5 commits May 11, 2017 09:54

fix resource check

e3a8744

improve comment

13d331a

add variable to toString

f217bc7

reset long running tasks per slave

d437ffe

dynamically set min offer score based on utilization across cluster

3e4d2e0

matush-v added the hs_qa label May 18, 2017

matush-v added 5 commits May 18, 2017 15:19

Merge branch 'master' into task-juggling

ec97c2d

handle invalid taskIds

d7077ac

split variables so it's more readable

df91216

treat Singularity Config as immutable

825761b

calculate min offer score based on cluster state

75cfd41

matush-v commented May 23, 2017

View reviewed changes

ssalinas reviewed May 23, 2017

View reviewed changes

don't save history usage summary data

1f6d650

matush-v commented May 23, 2017

View reviewed changes

matush-v added 3 commits May 24, 2017 14:17

calculate min score tolerance by number of slaves in cluster

5b43005

check for infinity, update tests

422a1af

increase tolerance -- shift log func up

c34834e

matush-v force-pushed the task-juggling branch from ee48576 to c34834e Compare May 26, 2017 17:19

calculate max possible task past due time based on task lag

3345776

matush-v commented May 31, 2017

View reviewed changes

matush-v added 2 commits June 1, 2017 10:49

don't wait for better offers

63624b3

maxTaskLagMillis -> maxTaskLag

c4b98ab

ssalinas added the hs_stable label Jun 2, 2017

ssalinas mentioned this pull request Jun 7, 2017

Combine offers to schedule tasks more efficiently #1561

Merged

ssalinas merged commit af223b1 into master Jun 8, 2017

ssalinas deleted the task-juggling branch June 8, 2017 16:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Better task balancing #1482

Better task balancing #1482

matush-v commented Mar 30, 2017 •

edited

Loading

matush-v Mar 30, 2017

ssalinas commented Mar 30, 2017

matush-v Apr 5, 2017

matush-v commented Apr 5, 2017

matush-v Apr 5, 2017

matush-v commented Apr 14, 2017

ssalinas Apr 19, 2017

matush-v Apr 20, 2017

ssalinas Apr 19, 2017

matush-v Apr 20, 2017

ssalinas Apr 19, 2017

ssalinas Apr 19, 2017

matush-v Apr 20, 2017

ssalinas Apr 20, 2017

ssalinas Apr 20, 2017

matush-v Apr 20, 2017

ssalinas Apr 20, 2017

ssalinas Apr 20, 2017

matush-v Apr 20, 2017

ssalinas Apr 20, 2017

matush-v May 23, 2017

ssalinas May 23, 2017

matush-v May 23, 2017

ssalinas May 23, 2017

matush-v May 23, 2017

matush-v May 23, 2017

matush-v commented May 23, 2017

matush-v May 31, 2017 •

edited

Loading

matush-v commented May 31, 2017

ssalinas commented Jun 8, 2017

Better task balancing #1482

Better task balancing #1482

Conversation

matush-v commented Mar 30, 2017 • edited Loading

Choose a reason for hiding this comment

ssalinas commented Mar 30, 2017

Choose a reason for hiding this comment

matush-v commented Apr 5, 2017

Before

After

Choose a reason for hiding this comment

matush-v commented Apr 14, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matush-v commented May 23, 2017

Before

After

matush-v May 31, 2017 • edited Loading

Choose a reason for hiding this comment

matush-v commented May 31, 2017

Before

After

ssalinas commented Jun 8, 2017

matush-v commented Mar 30, 2017 •

edited

Loading

matush-v May 31, 2017 •

edited

Loading