New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit the number of tasks that can shuffle for cpu rebalance at once #1751

Merged
merged 7 commits into from Mar 6, 2018

Conversation

Projects
None yet
3 participants
@ssalinas
Member

ssalinas commented Mar 1, 2018

Add a cluster-wide limit as well as a per-slave limit for how many tasks we can shuffle around at once

@baconmania

This comment has been minimized.

Contributor

baconmania commented Mar 1, 2018

🚢

ssalinas added some commits Mar 1, 2018

@@ -234,7 +246,12 @@ public void runActionOnPoll() {
if (slaveOverloaded && configuration.isShuffleTasksForOverloadedSlaves()) {
possibleTasksToShuffle.sort((u1, u2) -> Double.compare(u2.getUsage().getCpusUsed(), u1.getUsage().getCpusUsed()));
for (TaskIdWithUsage taskIdWithUsage : possibleTasksToShuffle) {
if (cpuOverage <= 0 || shuffledTasks > configuration.getMaxTasksToShuffleForCpuOverage()) {
if (requestsWithShuffledTasks.contains(taskIdWithUsage.getTaskId().getRequestId())) {
LOG.debug("Request {} laready has a shuffling task, skipping", taskIdWithUsage.getTaskId().getRequestId());

This comment has been minimized.

@baconmania

baconmania Mar 1, 2018

Contributor

nitpick typo :)

@baconmania

This comment has been minimized.

Contributor

baconmania commented Mar 1, 2018

🚢

@ssalinas ssalinas added the hs_stable label Mar 1, 2018

@pschoenfelder

This comment has been minimized.

Contributor

pschoenfelder commented Mar 5, 2018

🚢

@ssalinas

This comment has been minimized.

Member

ssalinas commented Mar 6, 2018

Going to merge this one and start other changes in a new PR

@ssalinas ssalinas merged commit 451716b into master Mar 6, 2018

1 of 2 checks passed

continuous-integration/travis-ci/pr The Travis CI build failed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@ssalinas ssalinas deleted the shuffle_limit branch Mar 6, 2018

@ssalinas ssalinas added this to the 0.20.0 milestone Mar 6, 2018

@ssalinas ssalinas modified the milestones: 0.20.0, 0.19.2 Mar 26, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment