Skip to content

Robust rollingupdate and rollback #1353

Closed

Description

@bgrant0607

In, PR #1325 we agreed rollingupdate should be factored out of kubecfg in the kubecfg overhaul.

#492 (comment) and PR #1007 discussed alternative approaches to rollingupdate.

What I'd recommend is that we implement a new rollingupdate library and a corresponding command-line wrapper. The library may be invoked by more general libraries/tools in the future.

The rolling update approach that should be used is that of creating a new replication controller with 1 replica, resizing the new (+1) and old (-1) controllers one by one, and then deleting the old controller once it reaches 0 replicas. Unlike the current approach, this predictably updates the set of pods regardless of unexpected failures.

The two replication controllers would need at least one differentiating label, which could use the image tag of the primary container of the pod, which is typically what motivates rolling updates.

It should be possible to apply the operation to services containing multiple release tracks, such as daily and weekly or canary and stable, by apply it to each track individually.

Activity

  1. added this to the v1.0 milestone on Oct 4, 2014
  2. alex-mohr commented on Nov 20, 2014

    @alex-mohr
    Contributor

    The approach described above would seem to work for stateless services and/or blue-green deployments, but I'm not sure how it would work for something like e.g. a redis master where the pod itself has storage it cares about.

    Are stateful pods a use case we want to support? If so, they would seem to require some form of in-place update?

  3. KyleAMathews commented on Nov 20, 2014

    @KyleAMathews
    Contributor

    I'd say yes with some sort of hook system to let people write custom migration scripts to support the transition say to the new Redis master.

  4. bgrant0607 commented on Nov 20, 2014

    @bgrant0607
    MemberAuthor

    @alex-mohr See #598 and #1515. You could also put the data into a PD.

  5. stp-ip commented on Nov 21, 2014

    @stp-ip
    Member

    The thing is we have to start somewhere and I would say we should start with a rolling update mechanism for stateless services only.

    Statefull services have a lot of different needs. I would try to avoid making k8s into a workflow integrating system. The question is. Would it be possible to let containers handle it themselves? On a shutdown lock and flush database. On restart unlock database etc. It could be handled on a per container basis, but makes them a lot more complex.
    Another idea, which only works for migrations in my opinion would be to spin up an intermediate pod.

    database example:
    -> let the container lock and flush database
    -> start intermediate container to migrate database
    -> remove intermediate container when finished (exit themself)
    -> start new pod one container at a time
    -> remove old pod one container at a time

    This would be the easiest example to start with. I agree there are other ways to remove the complexity from the containers themselves. This could be being able to execute a script on each old container via docker exec for example, which basically kills them when the script was executed. Either when all are killed the migration script could get in (downtime...) or the new containers could already be spawned.

  6. KyleAMathews commented on Nov 21, 2014

    @KyleAMathews
    Contributor

    @bgrant0607 looks like we'd need a pre-prestop though as SIGTERM is still sent to the container :)

  7. bgrant0607 commented on Jan 6, 2015

    @bgrant0607
    MemberAuthor

    Copying the detailed design of rolling update from #3061:

    Requirements, principles, assumptions, etc.:

    • As a matter of principle, any functionality available in kubectl should be available by library call and declaratively, as well as by command.
    • Rolling update should not mix configuration generation and orchestration of the update.
    • It needs to handle arbitrary (e.g., multi-container) pods.
    • Durable data objects are assumed to have independent lifetimes of the pods themselves.
    • If the client fails or is killed in the middle of the rolling update, there needs to be a reasonable way to recover and complete or rollback the rollout.
    • Users should be able to specify the update rate on the command line, especially given that we don't yet check pod health or readiness. If it just blazes through the pods, it might as well not be a rolling update.

    Proposed syntax:

    kubectl rollingupdate <old-controller-name>  -f <new-replication-controller.json>
    

    If the number of replicas specified in the new template were unspecified, I think it would default to 0 after parsing. Behavior in this case would be that we gradually increase it to the replica count of the original. We could also allow the user to specify a new size, which would do a rolling update for min(old,new) and then delete or add the remaining replicas.

    Since kubectl can reason about the type of the json parameter, we could add support for specifying just the pod template later (like when we have config generation and v1beta3 sorted), or for other types of controllers (e.g., per-node controller).

    Regarding recovery in the case of failure in the middle (i.e., resumption of the rolling update), and rollback of a partial rolling update: This is where annotations would be useful -- to store the final number of replicas desired, for instance. I think the identical syntax as above could work for recovery, if kubectl first checked whether the new replicationController already existed and, if so, merely continued. If we also supported just specifying a name for the new replication controller, that could also be used either to finish the rollingupdate or to rollback, by swapping old and new on the command line.

    We should use an approach friendly to resizing, either via kubectl or via an auto-scaler. We should keep track of the number of replicas subtracted in an annotation on the original replication controller, so that the total desired is the current replica count plus the number subtracted. Unless the user specified the desired number explicitly in the new replication controller -- that can be stored in an annotation on the new replication controller.

    I also eventually want to support "roll over" -- replace N replication controllers with one. I think that's straightforward if we just use the convention that the file or last name corresponds to the target replication controller, though it may be easier for users to shoot themselves in the foot. Perhaps a specific command-line argument to identify the new/target controller would therefore be useful.

    kubectl rollingupdate old1 old2 ... oldN new
    kubectl rollingupdate old1 old2 ... oldN -f new.json
    

    This issue should not be closed until we support recovery and rollover.

  8. davidopp commented on Jan 7, 2015

    @davidopp
    Contributor

    I don't understand why you need annotations. Isn't the rolling update essentially stateless, in the sense that you can figure out where you left off and what remains to be done just by looking at the old and new replication controllers and the pods?

  9. bgrant0607 commented on Jan 7, 2015

    @bgrant0607
    MemberAuthor

    Almost, but not quite. The 2 replication controllers are not changed atomically, so the count could be off by one without keeping track. It's also the case that we allow the size to be changed by the rolling update.

  10. changed the title [-]Robust rollingupdate[/-] [+]Robust rollingupdate and rollback[/+] on Jan 10, 2015
  11. bgrant0607 commented on Jan 14, 2015

    @bgrant0607
    MemberAuthor

    How to make rolling update friendly to auto-scaling is described here: #2863 (comment)

  12. 32 remaining items

  13. modified the milestones: , v1.0 on Apr 27, 2015
  14. added
    priority/backlogHigher priority than priority/awaiting-more-evidence.
    and removed
    priority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next release.
    on Apr 27, 2015
  15. bgrant0607 commented on Jul 17, 2015

    @bgrant0607
    MemberAuthor

    Updating from cli-roadmap.md:

    Deployment (#1743) will need rollover (replace multiple replication controllers with one) in order to kick off new rollouts as soon as the pod template is updated.

    We should think about whether we still want annotations on the underlying replication controllers and, if so, whether they need to be improved: #2863 (comment)

  16. removed this from the milestone on Jul 24, 2015
  17. added
    priority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next release.
    and removed
    priority/backlogHigher priority than priority/awaiting-more-evidence.
    on Jul 27, 2015
  18. bgrant0607 commented on Jul 27, 2015

    @bgrant0607
    MemberAuthor

    I believe we have more specific issues filed for remaining work, so I'm closing this "metaphysical" issue.

  19. huang195 commented on Dec 7, 2015

    @huang195
    Contributor

    @grant0607 is readiness probe and/or liveness probe taken into account of rolling update? What is the expected behavior when these probes are in good/bad conditions?

  20. added a commit that references this issue on Sep 1, 2026
    a35a53b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/app-lifecyclearea/kubectlpriority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next release.sig/api-machineryCategorizes an issue or PR as relevant to SIG API Machinery.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions