Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incremental deploys #817

Merged
merged 46 commits into from Feb 23, 2016
Merged

Incremental deploys #817

merged 46 commits into from Feb 23, 2016

Conversation

ssalinas
Copy link
Member

@ssalinas ssalinas commented Dec 21, 2015

Still needs plenty of work, but opening up early for feedback.

Right now I am adding a field to the SingularityPendingDeploy object that will track the progress of an incremental deploy as it goes along. This new field is an object that holds information about the current target number of instances and rate at which the deploy should proceed.

Still to-do:

  • Better scheduling of upcoming tasks. Right now it's just keying off of numMissingInstances, but since we know the time between deploy steps, we can go ahead and schedule them
  • Make it play nice with load balancer add/remove. Regular deploys add everything all at once when the deploy succeeds, this will need to do it per-task
  • Surface more information about what is happening in the deploy, if you aren't watching the logs it's kind of mysterious right now
  • Option to manually move the deploy forward or backward a step (should be fairly straightforward since we have the info in the SingularityPendingDeploy object)
  • Tests, tests, tests, tests, tests....
  • UI

/cc @tpetr @wsorenson

@@ -0,0 +1,5 @@
package com.hubspot.singularity;

public enum DeployStyle {
Copy link
Contributor

@tpetr tpetr Dec 21, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't want to overcomplicate things, but I wonder about the merits of an enum vs. being able to define a batch size for the deploy that defaults to the total instance count (i.e. a "normal" deploy)

Copy link
Member Author

@ssalinas ssalinas Dec 21, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that's fair, might keep it simpler in the end, move towards all deploys working the same, but 'regular' deploys are just everything at once

@tpetr tpetr added this to the 0.4.7 milestone Dec 22, 2015
@ssalinas ssalinas force-pushed the incremental_deploys branch 4 times, most recently from 50f69f9 to d5dc040 Compare Dec 24, 2015
@tpetr tpetr modified the milestones: 0.4.7, 0.4.8 Dec 28, 2015
@ssalinas
Copy link
Member Author

ssalinas commented Dec 29, 2015

The basics are working now. All deploys now use a similar system where progress is tracked in the pending deploy object, incrementing a target number of instances for each step of the deploy. By default the number of instances per deploy step is the total instances (i.e. everything at once like before) unless a deployRate is set that then determines how many instances to deploy in each 'step'. Load balancer actions are triggered at the end of each step.

Some info is surfaced in the ui now on the request page, but the wording/design could probably still be a bit cleaner.

@@ -169,7 +177,10 @@ public SingularityDeployBuilder toBuilder() {
.setEnv(copyOfMap(env))
.setUris(copyOfList(uris))
.setExecutorData(executorData)
.setLabels(labels);
.setLabels(labels)

Copy link
Contributor

@tpetr tpetr Dec 29, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^ can you remove the newline? can get confusing in a builder block

@@ -50,6 +51,7 @@
@Api(description="Manages Singularity Deploys for existing requests", value=DeployResource.PATH, position=2)
public class DeployResource extends AbstractRequestResource {
public static final String PATH = SingularityService.API_BASE_PATH + "/deploys";
public static final int DEFAULT_DEPLOY_STEP_WAIT_TIME_SECONDS = 60;
Copy link
Member Author

@ssalinas ssalinas Dec 29, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tpetr was debating making the default wait time 0, would that make more sense? i.e. move on to the next step as soon as the previous is finished 'rolling deploy', by default

Copy link
Contributor

@tpetr tpetr Dec 29, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah i think that jives with the incremental deploy idea

@ssalinas ssalinas mentioned this pull request Feb 9, 2016
@ssalinas
Copy link
Member Author

ssalinas commented Feb 9, 2016

@tpetr @wsorenson finished a bunch of refactoring on this to slim down the methods. Also added more tests for canceled/failed deploy scenarios. Let me know if you see anything else in here that's needs updating

return targetActiveInstances;
}

@Override public String toString() {
Copy link
Contributor

@tpetr tpetr Feb 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a recent intellij thing that started happening, might not be a setting that the eclipse formatter file sets. i think it'd be good to keep all our @Override's on their own line

@tpetr tpetr modified the milestones: 0.4.10, 0.4.11 Feb 18, 2016
@ssalinas ssalinas modified the milestones: 0.4.11, 0.4.12 Feb 23, 2016
ssalinas added a commit that referenced this pull request Feb 23, 2016
@ssalinas ssalinas merged commit 0adc4dc into master Feb 23, 2016
2 checks passed
@ssalinas ssalinas deleted the incremental_deploys branch Feb 23, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants