New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better task balancing #1482

Merged
merged 73 commits into from Jun 8, 2017
Commits
Jump to file or symbol
Failed to load files and symbols.
+28 鈭4
Diff settings

Always

Just for now

Viewing a subset of changes. View all

consider time overdue and offers passed when picking best offer

  • Loading branch information...
darcatron committed Apr 5, 2017
commit f90bb4b331272d04c6905429a3b59b2b234e2980
@@ -6,6 +6,7 @@
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
import javax.inject.Singleton;
@@ -57,6 +58,8 @@
private final Provider<SingularitySchedulerStateCache> stateCacheProvider;
private final SchedulerDriverSupplier schedulerDriverSupplier;
private final Map<String, Integer> offerMatchAttemptsPerTask = new HashMap<>();
@Inject
public SingularityMesosOfferScheduler(MesosConfiguration mesosConfiguration,
CustomExecutorConfiguration customExecutorConfiguration,
@@ -122,7 +125,7 @@ public SingularityMesosOfferScheduler(MesosConfiguration mesosConfiguration,
for (SingularityTaskRequestHolder taskRequestHolder : pendingTaskIdToTaskRequest.values()) {
Map<SingularityOfferHolder, Double> scorePerOffer = new HashMap<>();
double minScore = minScore(taskRequestHolder.getTaskRequest());
for (SingularityOfferHolder offerHolder : offerHolders) {
if (configuration.getMaxTasksPerOffer() > 0 && offerHolder.getAcceptedTasks().size() >= configuration.getMaxTasksPerOffer()) {
@@ -131,12 +134,15 @@ public SingularityMesosOfferScheduler(MesosConfiguration mesosConfiguration,
}
double score = score(offerHolder, stateCache, tasksPerOfferPerRequest, taskRequestHolder, getSlaveUsage(currentSlaveUsages, offerHolder.getOffer().getSlaveId().getValue()));

This comment has been minimized.

@ssalinas

ssalinas Apr 20, 2017

Member

for clarity, maybe something like 'hostScore' here? The score is for the particular slave, not necessarily about the offer

@ssalinas

ssalinas Apr 20, 2017

Member

for clarity, maybe something like 'hostScore' here? The score is for the particular slave, not necessarily about the offer

This comment has been minimized.

@darcatron

darcatron Apr 20, 2017

Contributor

I'm not sure about the naming here. We do look at the slave's utilization to score the offer, but we are still scoring the offer itself since offers aren't uniquely 1:1 for a slave (e.g. 2 offers for the same slave).

The slave utilization weight will be the same for all offers on the same slave, but the offer resources will be different per offer. So, it seems to me that we're scoring the offer in this class rather than the slave itself

@darcatron

darcatron Apr 20, 2017

Contributor

I'm not sure about the naming here. We do look at the slave's utilization to score the offer, but we are still scoring the offer itself since offers aren't uniquely 1:1 for a slave (e.g. 2 offers for the same slave).

The slave utilization weight will be the same for all offers on the same slave, but the offer resources will be different per offer. So, it seems to me that we're scoring the offer in this class rather than the slave itself

if (score > 0) {
// todo: can short circuit here if score is high enough
if (score >= minScore) {
// todo: can short circuit here if score is high enough (>= .9)
scorePerOffer.put(offerHolder, score);

This comment has been minimized.

@darcatron

darcatron Mar 30, 2017

Contributor

Thought we might want to have a value that's definitely good enough to just accept instead of continue evaluating

@darcatron

darcatron Mar 30, 2017

Contributor

Thought we might want to have a value that's definitely good enough to just accept instead of continue evaluating

}
}
offerMatchAttemptsPerTask.compute(taskRequestHolder.getTaskRequest().getPendingTask().getPendingTaskId().getId(),
(k, v) -> (scorePerOffer.isEmpty() ? (v == null ? offerHolders.size() : v + offerHolders.size()) : null));
if (!scorePerOffer.isEmpty()) {
SingularityOfferHolder bestOffer = Collections.max(scorePerOffer.entrySet(), Map.Entry.comparingByValue()).getKey();
LOG.info("Best offer is {} with a score of {}/1", bestOffer, scorePerOffer.get(bestOffer));
@@ -273,7 +279,25 @@ private double score(Offer offer, SingularityTaskRequest taskRequest, Optional<S
return score;
}
private SingularityTask acceptTask(SingularityOfferHolder offerHolder, SingularitySchedulerStateCache stateCache, Map<String, Map<String, Integer>> tasksPerOfferPerRequest, SingularityTaskRequestHolder taskRequestHolder) {
private double minScore(SingularityTaskRequest taskRequest) {
double minScore = 0.80;

This comment has been minimized.

@darcatron

darcatron Apr 5, 2017

Contributor

this can be adjusted as necessary. I thought an 80% match might be a good starting point, but we could def reduce it

@darcatron

darcatron Apr 5, 2017

Contributor

this can be adjusted as necessary. I thought an 80% match might be a good starting point, but we could def reduce it

int maxOfferAttempts = 20;

This comment has been minimized.

@ssalinas

ssalinas Apr 20, 2017

Member

another that would be nice to have ocnfigurable

@ssalinas

ssalinas Apr 20, 2017

Member

another that would be nice to have ocnfigurable

long maxMillisPastDue = TimeUnit.MILLISECONDS.convert(10, TimeUnit.MINUTES);
minScore -= offerMatchAttemptsPerTask.getOrDefault(taskRequest.getPendingTask().getPendingTaskId().getId(), 0) / maxOfferAttempts;
minScore -= millisPastDue(taskRequest, System.currentTimeMillis()) / maxMillisPastDue;
return Math.max(minScore, 0);
}
private long millisPastDue(SingularityTaskRequest taskRequest, long now) {
return Math.max(now - taskRequest.getPendingTask().getPendingTaskId().getNextRunAt(), 1);
}
private SingularityTask acceptTask(SingularityOfferHolder offerHolder,
SingularitySchedulerStateCache stateCache,
Map<String, Map<String, Integer>> tasksPerOfferPerRequest,
SingularityTaskRequestHolder taskRequestHolder) {
final SingularityTaskRequest taskRequest = taskRequestHolder.getTaskRequest();
final SingularityTask task = mesosTaskBuilder.buildTask(offerHolder.getOffer(), offerHolder.getCurrentResources(), taskRequest, taskRequestHolder.getTaskResources(), taskRequestHolder.getExecutorResources());
ProTip! Use n and p to navigate between commits in a pull request.