New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better task balancing #1482

Merged
merged 73 commits into from Jun 8, 2017
Commits
Jump to file or symbol
Failed to load files and symbols.
+12 鈭4
Diff settings

Always

Just for now

Viewing a subset of changes. View all

better handling of missing usage, improved logging

  • Loading branch information...
darcatron committed May 2, 2017
commit e24249273f47a47dd6d08878e1fc2b0e3e5a8d28
@@ -137,7 +137,7 @@ public SingularityMesosOfferScheduler(MesosConfiguration mesosConfiguration,
Map<SingularityOfferHolder, Double> scorePerOffer = new HashMap<>();
double minScore = minScore(taskRequestHolder.getTaskRequest(), offerMatchAttemptsPerTask, System.currentTimeMillis());
LOG.trace("Minimum score for task {} is {}", taskRequestHolder.getTaskRequest().getPendingTask().getPendingTaskId().getId(), minScore);
LOG.trace("Minimum score {} for task {}", minScore, taskRequestHolder.getTaskRequest().getPendingTask().getPendingTaskId().getId());
for (SingularityOfferHolder offerHolder : offerHolders) {
@@ -147,7 +147,7 @@ public SingularityMesosOfferScheduler(MesosConfiguration mesosConfiguration,
}
double score = score(offerHolder, stateCache, tasksPerOfferPerRequest, taskRequestHolder, getSlaveUsage(currentSlaveUsages, offerHolder.getOffer().getSlaveId().getValue()));

This comment has been minimized.

@ssalinas

ssalinas Apr 20, 2017

Member

for clarity, maybe something like 'hostScore' here? The score is for the particular slave, not necessarily about the offer

@ssalinas

ssalinas Apr 20, 2017

Member

for clarity, maybe something like 'hostScore' here? The score is for the particular slave, not necessarily about the offer

This comment has been minimized.

@darcatron

darcatron Apr 20, 2017

Contributor

I'm not sure about the naming here. We do look at the slave's utilization to score the offer, but we are still scoring the offer itself since offers aren't uniquely 1:1 for a slave (e.g. 2 offers for the same slave).

The slave utilization weight will be the same for all offers on the same slave, but the offer resources will be different per offer. So, it seems to me that we're scoring the offer in this class rather than the slave itself

@darcatron

darcatron Apr 20, 2017

Contributor

I'm not sure about the naming here. We do look at the slave's utilization to score the offer, but we are still scoring the offer itself since offers aren't uniquely 1:1 for a slave (e.g. 2 offers for the same slave).

The slave utilization weight will be the same for all offers on the same slave, but the offer resources will be different per offer. So, it seems to me that we're scoring the offer in this class rather than the slave itself

LOG.trace("Offer {} with resources {} scored {} for Task {}", offerHolder.getOffer(), offerHolder.getCurrentResources(), score, taskRequestHolder.getTaskRequest().getPendingTask().getPendingTaskId().getId());
LOG.trace("Scored {} for task {} with offer for slave {} and resources {} ", score, taskRequestHolder.getTaskRequest().getPendingTask().getPendingTaskId().getId(), offerHolder.getOffer().getSlaveId(), offerHolder.getCurrentResources());
if (score != 0 && score >= minScore) {
// todo: can short circuit here if score is high enough (>= .9)
@@ -266,8 +266,8 @@ private double score(SingularityOfferHolder offerHolder, SingularitySchedulerSta
@VisibleForTesting
double score(Offer offer, SingularityTaskRequest taskRequest, Optional<SingularitySlaveUsageWithId> maybeSlaveUsage) {

This comment has been minimized.

@ssalinas

ssalinas Apr 20, 2017

Member

Let's go over this one in-person, think we are getting close, just easier to chat than typing a novel in github ;)

@ssalinas

ssalinas Apr 20, 2017

Member

Let's go over this one in-person, think we are getting close, just easier to chat than typing a novel in github ;)

if (!maybeSlaveUsage.isPresent() || !maybeSlaveUsage.get().getCpuTotal().isPresent() || !maybeSlaveUsage.get().getMemoryMbTotal().isPresent()) {
LOG.info("Slave {} has no total usage data. Will default to {}", offer.getSlaveId().getValue(), configuration.getDefaultOfferScoreForMissingUsage());
if (isMissingUsageData(maybeSlaveUsage)) {
LOG.info("Slave {} has missing usage data ({}). Will default to {}", offer.getSlaveId().getValue(), maybeSlaveUsage, configuration.getDefaultOfferScoreForMissingUsage());
return configuration.getDefaultOfferScoreForMissingUsage();
}
@@ -284,6 +284,14 @@ private double score(SingularityOfferHolder offerHolder, SingularitySchedulerSta
: scoreNonLongRunningTask(taskRequest, longRunningMemUsedScore, memTotalScore, longRunningCpusUsedScore, cpusTotalScore);
}
private boolean isMissingUsageData(Optional<SingularitySlaveUsageWithId> maybeSlaveUsage) {
return !maybeSlaveUsage.isPresent() ||
!maybeSlaveUsage.get().getCpuTotal().isPresent() || !maybeSlaveUsage.get().getMemoryMbTotal().isPresent() ||
maybeSlaveUsage.get().getLongRunningTasksUsage() == null ||
!maybeSlaveUsage.get().getLongRunningTasksUsage().containsKey(ResourceUsageType.CPU_USED) ||
!maybeSlaveUsage.get().getLongRunningTasksUsage().containsKey(ResourceUsageType.MEMORY_BYTES_USED);
}
private boolean isLongRunning(SingularityTaskRequest taskRequest) {
return taskRequest.getRequest().getRequestType().isLongRunning();
}
ProTip! Use n and p to navigate between commits in a pull request.