Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle scale during deploy #1037

Merged
merged 5 commits into from May 17, 2016
Merged

Handle scale during deploy #1037

merged 5 commits into from May 17, 2016

Conversation

@ssalinas
Copy link
Member

@ssalinas ssalinas commented May 13, 2016

@stevenschlansker should fix that edge case you ran into the other day when scaling + deploying at the same time

@ssalinas ssalinas added this to the 0.6.0 milestone May 13, 2016
@tpetr
Copy link
Member

@tpetr tpetr commented May 13, 2016

Could you add a unit test for this?

@ssalinas
Copy link
Member Author

@ssalinas ssalinas commented May 13, 2016

test added

@@ -603,7 +603,7 @@ private SingularityDeployResult markStepFinished(SingularityPendingDeploy pendin
int numTasksToShutDown = Math.max(otherActiveTasks.size() - (request.getInstancesSafe() - deployProgress.getTargetActiveInstances()), 0);
List<SingularityTaskId> sortedOtherTasks = new ArrayList<>(otherActiveTasks);
Collections.sort(sortedOtherTasks, SingularityTaskId.INSTANCE_NO_COMPARATOR);
return sortedOtherTasks.subList(0, numTasksToShutDown);
return sortedOtherTasks.subList(0, Math.min(numTasksToShutDown, sortedOtherTasks.size() - 1));

This comment has been minimized.

@tpetr

tpetr May 13, 2016
Member

i might be misunderstanding something obvious, but why subtract one here?

This comment has been minimized.

@ssalinas

ssalinas May 13, 2016
Author Member

e.g. size of the array is 2, so the max index we can use for the sublist is 1

This comment has been minimized.

@tpetr

tpetr May 13, 2016
Member

But if you look at that javadoc, toIndex is exclusive:

     * @param fromIndex low endpoint (inclusive) of the subList
     * @param toIndex high endpoint (exclusive) of the subList
     * @return a view of the specified range within this list
     * @throws IndexOutOfBoundsException for an illegal endpoint index value
     *         (<tt>fromIndex &lt; 0 || toIndex &gt; size ||
     *         fromIndex &gt; toIndex</tt>)
     */

This comment has been minimized.

@ssalinas

ssalinas May 13, 2016
Author Member

So, is it the case that the index out of bounds is coming from the from index then instead? Had just assumed it was on the high end when I made this PR. must have been that the list was empty, will fix, thanks

This comment has been minimized.

@ssalinas

ssalinas May 13, 2016
Author Member

Fxed to check both for toIndex errors(with the Math.min) and fromIndex (isEmpty check). First condition gets hit by the test I added, second condition will get hit by a number of other tests

@ssalinas ssalinas added the hs_qa label May 16, 2016
@ssalinas ssalinas added the hs_stable label May 16, 2016
@ssalinas
Copy link
Member Author

@ssalinas ssalinas commented May 17, 2016

Found one last hitch with scale during deploy. If the timing is right, you will end up with the pre-scale number of requests running after the deploy is finished. The deploy ends with the targetActiveInstances, but we currently don't enqueue another pending request on successful deploy. Going to extend the test to check for this case then figure out the best way to fix it

@ssalinas ssalinas changed the title avoid index out of bounds when finding tasks to shut down Handle scale during deploy May 17, 2016
@tpetr
Copy link
Member

@tpetr tpetr commented May 17, 2016

LGTM

@ssalinas ssalinas merged commit 4a3fdb8 into master May 17, 2016
0 of 2 checks passed
0 of 2 checks passed
continuous-integration/travis-ci/pr The Travis CI build failed
Details
continuous-integration/travis-ci/push The Travis CI build failed
Details
@ssalinas ssalinas deleted the avoid_oob branch May 17, 2016
@ssalinas ssalinas removed hs_qa labels May 17, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.