New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature request - rolling version update for apps #1829

Closed
PierreKircher opened this Issue Sep 13, 2014 · 13 comments

Comments

Projects
None yet
8 participants
@PierreKircher
Contributor

PierreKircher commented Sep 13, 2014

at some point you need to test your app with real traffic

you got v5 deployed and its stable now works ok and now you need to create the next killer feature ..

feature ready .. testing done .. (but nothing tests as good as real traffic .. !)

.. v6 .. but to see if anything behaves correctly you need to have live traffic ..

100% direct traffic could cost you a lot in a given scenario ..
of course you can rollback but you still affect 100% of your user-base

so a rolling update function
for balancing would be nice ..
lets say for the next N hours (timescale should be "adjustable" i am going to use %) gradually update traffic towards v6

why ? simple .. risks management ( cost of fail )
V5 V6
10%| 90 10 - ( get some peaks with a few customers .. - easy to spot errors ..)
20%| 80 20 -
30%| 70 30 -
40%| 60 40 -
50%| 50 50 - ( decent amount of traffic get realistic insight )
60%| 40 60 -
70%| 30 70 -
80%| 20 80 - ( nearly there .. if anything goes wrong we should have spotted it by now )
90%| 10 90 -
100 full deploy of v6 kill - v5 .. ( we are able to restore it anyway )

if something errors in that timescale .. you could simply rollback to 100% v5 .. fix your problem and deploy v5 to v7 with the same mechanics ..

HOW ?

we use nginx as routing mechanics anyway ( as of now )..

weight=number
sets the weight of the server, by default, 1.

upstream XXX {
server v5.example.com weight=9;
server v6.example.com weight=1;
}

basically what we need is a way to tell to not kill the older versions and keep it around .. to verify

deis config:set GRADUAL_DEPLOY=Xm/h

any ideas / insights would be appreciated

@PierreKircher PierreKircher changed the title from feature request - rolling version upgrade to feature request - rolling version update Sep 13, 2014

@ianblenke

This comment has been minimized.

ianblenke commented Sep 13, 2014

+1 for rolling releases.

The big challenge with synchronizing your database schema with your code releases is having the code-paths continue the "old" way until you are 100% on the new codebase, and then throw a configuration switch (typically in the database) that enables a new code-paths that use the new schema.

@PierreKircher

This comment has been minimized.

Contributor

PierreKircher commented Sep 13, 2014

this feature is intended to be optionally .. not forced / not for everyone .. of course db-schema is a problem but that comes down to convention

@ianblenke

This comment has been minimized.

ianblenke commented Sep 13, 2014

There are numerous corner cases from this. The first notable one is that overlapping deploys of new version happening within the same time window might trample on one another. That means putting up fences to prevent more deploys so as not to interfere with an on-going gradual deploy. If the gradual deploy gets "stuck", there would need to be a way to override/stop that gradual deploy in whatever state it is in and tear down the fence to allow new deploys to happen.

@PierreKircher

This comment has been minimized.

Contributor

PierreKircher commented Sep 13, 2014

agree 100% .. probably a cleanup service to enable that .. like google's mosquito

@edude03

This comment has been minimized.

edude03 commented Sep 14, 2014

I know the deis team is (was?) looking at replacing nginx with vulcand which actually supports this sort of use case

@carmstrong

This comment has been minimized.

Contributor

carmstrong commented Sep 14, 2014

This would be great to have tooling around rolling/canary deploys, but I think it would be pretty application-specific (i.e. when do you feel safe with the next deployment phase? what is performant for your use case?)

You could theoretically do this now with your own tooling (Jenkins or something) which would do a deis scale 5 and then 10, etc, as you feel comfortable. We are already planning to implement some sort of metrics integration (#981), and that could feed into something like this.

This isn't on our roadmap for stable, but I'd like to think about how it makes sense in Deis core.

@carmstrong

This comment has been minimized.

Contributor

carmstrong commented Sep 14, 2014

Also potentially related - currently, not cleaning up old app releases when you deploy is currently a bug: #1149. So, allowing multiple versions of an app to run would be a requisite feature request.

@carmstrong

This comment has been minimized.

Contributor

carmstrong commented Sep 14, 2014

Additionally, having a pipelines feature would allow testing of a build in a separate environment (dev, test, etc.) and then promoting it to production once it's been tested: #1318

@PierreKircher

This comment has been minimized.

Contributor

PierreKircher commented Sep 14, 2014

there is a set of factors you cant test in stage and you are required to have real traffic on it .. thats why i asked for it in the first place

@bacongobbler

This comment has been minimized.

Member

bacongobbler commented Sep 15, 2014

@edude03 vulcand is missing several important features over nginx, most prominent being tcp support. Nginx has support for rolling upgrades. We just have to figure out a way to handle this properly.

@gabrtv gabrtv removed the production label Oct 10, 2014

@gebrits

This comment has been minimized.

gebrits commented Nov 9, 2014

This is probably incredibly naive (just reading up today on Deis and other cluster orchestration tools), but perhaps using ConfD would help to set this up as discussed for instance in this blog post?

@bacongobbler

This comment has been minimized.

Member

bacongobbler commented Nov 10, 2014

@gebrits our router uses confd. IIRC we were the ones who supplied the initial documentation around using nginx with confd. We're heavy users of it

@carmstrong carmstrong changed the title from feature request - rolling version update to feature request - rolling version update for apps May 12, 2015

@mboersma

This comment has been minimized.

Member

mboersma commented Nov 3, 2015

Closing in favor of #3685.

@mboersma mboersma closed this Nov 3, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment