Write proposal for controller pod management: adoption, orphaning, ownership, etc. (aka controllers v2)

[bgrant0607]

Write a comprehensive proposal for how controllers should manage sets of pods. The main goal is to make controller APIs more usable and less error-prone.

We've discussed a number of changes:

• unique label generation: Generate unique label for Replication Controller, Job #12298, Server side defaulting of values prevents diff patches from working #34292
• controllerRef to manage overlap: Validate no replicationController overlap. #2210 (comment)
• server-side cascading deletion via existence dependences: Consistently support graceful and immediate termination for all objects #1535 (comment)
• GC prevention via finalizers: Places for hooks #3585
• deletion of non-conforming pods: https://github.com/kubernetes/kubernetes/blob/master/docs/design/daemon.md#cluster-mutations
• generation/observedGeneration: Create per-object sequence number and report last value seen in status of each object #7328
• status field containing selector query: Links to related object in json responses #3676
• conditions: Add Conditions alongside Phases. #14181
• aggregate status: Add status column for kubectl get replicationcontroller/service #7483
• policy for naming generated resources: ReplicationController pod template ignores generateName (ignores all ObjectMeta in fact) #15776
• reverse label lookup: Reverse lookup by labels #1348

We may want to split the following into separate issues:

Changes that would facilitate static work/role assignment:

• identity assignment: PetSet (was nominal services) #260, Indexed Job #14188
• PVC replication: Proposal: Allow anonymous inline claim in pod definition #12450

Long-standing idea to improve security and reusability around templates:

• separate template: Separate the pod template from replicationController #170

Reusability could also be addressed by:

• template proposal: initial template and parameterization proposal #18215

Also need to make it easier to update existing pods:

• For labels, currently need to update pod template, then pods, then selector: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/rolling_updater.go#L616 . The loop to update pods after that wouldn't be necessary if it watched observedGeneration (Create per-object sequence number and report last value seen in status of each object #7328).
• Inline updates: In-place rolling updates #9043
• vertical auto-scaling

[bgrant0607]

A wrinkle: updating a selector in Deployment #14894

[bgrant0607]

Some use cases for orphaning and/or adoption of pods:

• Stop the controller and restart it later. For instance, that's the way we pause Deployment at the moment.
• Update some attributes of the controller that can't be updated by deleting and re-creating it. For instance, renaming the controller, which we do in "simple rolling update".
• Adoption by an non-backward-compatible API resource.
• Update labels on existing pods that would be incompatible with the controller. Delete controller, change pod labels, create new controller with new selector.
• Bootstrapping. Create pods to start the control plane, then create their controllers.
• System exploration/learning. Create a pod, then a controller to manage it.
• Debugging. Change a pod's labels to orphan it so that it can be replaced and debugged out of the critical path.
• Replacement. Replace a pod generated from the template with a special one, perhaps with special support for profiling, tracing, audit, monitoring, etc.

[bgrant0607]

If we had an imperative API, a controllerRef backpointer could be manipulated to transfer ownership: transfer pods from controller X to controller Y. Orphaning would be somewhat weird in that a dangling controllerRef pointer would need to be left. Otherwise, there would be no attribute to select on for re-adoption. Or one could require transferring ownership before deleting the previous controller.

With our declarative API, we can't get rid of the labels and selector since those identify which pods should be adopted -- there needs to be some known unique set of attributes to select on.

Will also include nominal services #260, splitting out the template #170, and nominal jobs #14188 in the proposal. See also #12450.

Ref #8190, since this is a "next-gen" API proposal.

[bgrant0607]

Demo of Kelsey removing a label in order to orphan a pod for debugging: https://www.youtube.com/watch?feature=player_detailpage&v=-8aUxpVrD40#t=2193
https://github.com/kelseyhightower/yapc-asia-2015/blob/master/demo/README.md#troubleshooting

[pmorie]

@kubernetes/rh-cluster-infra

[davidopp]

If we had an imperative API, a controllerRef backpointer could be manipulated to transfer ownership: transfer pods from controller X to controller Y. Orphaning would be somewhat weird in that a dangling controllerRef pointer would need to be left. Otherwise, there would be no attribute to select on for re-adoption. Or one could require transferring ownership before deleting the previous controller.

With our declarative API, we can't get rid of the labels and selector since those identify which pods should be adopted -- there needs to be some known unique set of attributes to select on.

Maybe it's a semantic quibble, but I don't think the issue is imperative vs. declarative -- it's implicit vs. explicit encoding of which controller owns a pod. I see the as both being declarative.

Do we have any examples/uses cases of adoption yet?

I think it would be great if we could allow people to write controllers that only support the explicit model, i.e. don't have a Selector. (The controller would fill in controllerRef for the pods it creates.)

[bgrant0607]

By imperative, I meant operations like "transfer pods from controller X to controller Y".

Adoption requests: #11209
Other scenarios are described above.

[nikhiljindal]

/sub

[bgrant0607]

cc @smarterclayton

[soltysh]

/sub

[davidopp]

Having a backpointer from pod to controller will greatly simplify scheduler code, consider for example the hoops we jump through to determine if a pod is controlled by a particular RC in CalculateSpreadPriority() and CalculateAntiAffinityPriority() (both in selector_spreading.go). A backpointer to the service would also be nice (would simplify CalculateSpreadPriority(), which spreads on both RC and service).

[smarterclayton]

Back pointer to service has other undesirable characteristics - like
services have to mutate pods to reference them, which means services
controller is another thing which has effective root access to the
cluster. Is the benefit high enough?

On Dec 1, 2015, at 5:28 AM, David Oppenheimer notifications@github.com
wrote:

Having a backpointer from pod to controller will greatly simplify scheduler
code, consider for example the hoops we jump through to determine if a pod
is controlled by a particular RC in CalculateSpreadPriority() and
CalculateAntiAffinityPriority() (both in selector_spreading.go). A
backpointer to the service would also be nice (would simplify
CalculateSpreadPriority(), which spreads on both RC and service).

—
Reply to this email directly or view it on GitHub
#14961 (comment)
.

[bgrant0607]

Template proposal: #18215

PetSet proposal: #18016

[bgrant0607]

For 1.3, we're working on:

• controllerRef
• cascading deletion
• finalizers
• generation/observedGeneration
• PetSet
• Templates

[bgrant0607]

Note that unique label/selector generation similar to what we do for Job will require changes to kubectl expose, if will want users to be able to do a rolling update to an RC/RS with auto-generated labels.

#17902
https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/kubernetes-dev/WbqQVNkDZUE/1f5WZK2zCAAJ

[0xmichalis]

Make controllers aware of namespace termination: #38612

[bgrant0607]

v1 plan: #42752

cc @kow3ns

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[soltysh]

/remove-lifecycle stale
/lifecycle frozen