Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit the number of tasks that can shuffle for cpu rebalance at once #1751

Merged
merged 7 commits into from Mar 6, 2018

Conversation

@ssalinas
Copy link
Member

@ssalinas ssalinas commented Mar 1, 2018

Add a cluster-wide limit as well as a per-slave limit for how many tasks we can shuffle around at once

@baconmania
Copy link
Contributor

@baconmania baconmania commented Mar 1, 2018

🚢

@@ -234,7 +246,12 @@ public void runActionOnPoll() {
if (slaveOverloaded && configuration.isShuffleTasksForOverloadedSlaves()) {
possibleTasksToShuffle.sort((u1, u2) -> Double.compare(u2.getUsage().getCpusUsed(), u1.getUsage().getCpusUsed()));
for (TaskIdWithUsage taskIdWithUsage : possibleTasksToShuffle) {
if (cpuOverage <= 0 || shuffledTasks > configuration.getMaxTasksToShuffleForCpuOverage()) {
if (requestsWithShuffledTasks.contains(taskIdWithUsage.getTaskId().getRequestId())) {
LOG.debug("Request {} laready has a shuffling task, skipping", taskIdWithUsage.getTaskId().getRequestId());

This comment has been minimized.

@baconmania

baconmania Mar 1, 2018
Contributor

nitpick typo :)

@baconmania
Copy link
Contributor

@baconmania baconmania commented Mar 1, 2018

🚢

@ssalinas ssalinas added the hs_stable label Mar 1, 2018
@pschoenfelder
Copy link
Contributor

@pschoenfelder pschoenfelder commented Mar 5, 2018

🚢

@ssalinas
Copy link
Member Author

@ssalinas ssalinas commented Mar 6, 2018

Going to merge this one and start other changes in a new PR

@ssalinas ssalinas merged commit 451716b into master Mar 6, 2018
1 of 2 checks passed
1 of 2 checks passed
continuous-integration/travis-ci/pr The Travis CI build failed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
@ssalinas ssalinas deleted the shuffle_limit branch Mar 6, 2018
@ssalinas ssalinas added this to the 0.20.0 milestone Mar 6, 2018
@ssalinas ssalinas modified the milestones: 0.20.0, 0.19.2 Mar 26, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants