Additional pod life-cycle hook to support commit+push after container exits

[derekwaynecarr]

I would like the ability to define a hook to commit a container, and push its resulting image to a registry after its completed execution on the local node via a new policy on the Pod.

The use case is for a run-once pod whose result represents an image that should be pushed to a registry for consumption by other actors in the system. To do this today, your container needs access to the docker socket to launch its own container, do the commit, and push outside the knowledge of the Kubelet/Scheduler unaware of actual capacity on the node.

Want to open the issue for discussion, and if it has merit from community, will look to move to a design proposal.

/cc @smarterclayton @ncdc @csrwng @bparees
/cc @kubernetes/goog-control-plane @kubernetes/goog-node

[smarterclayton]

What credentials are used for the push? Is this generalizable to commit
and not push? Is this specific to a certain container runtime? Is this a
pod lifecycle behavior that could potentially be described in extensible
terms? Also, doesn't this assume that the container may need to be set up
with inputs by another container first (stdin via attach or volume mounts)?

On Fri, Sep 25, 2015 at 11:06 AM, Derek Carr notifications@github.com
wrote:

I would like the ability to define a hook to commit a container, and push
its resulting image to a registry after its completed execution on the
local node via a new policy on the Pod.

The use case is for a run-once pod whose result represents an image that
should be pushed to a registry for consumption by other actors in the
system. To do this today, your container needs access to the docker socket
to launch its own container, do the commit, and push outside the knowledge
of the Kubelet/Scheduler unaware of actual capacity on the node.

Want to open the issue for discussion, and if it has merit from community,
will look to move to a design proposal.

/cc @smarterclayton https://github.com/smarterclayton @ncdc
https://github.com/ncdc @csrwng https://github.com/csrwng @bparees
https://github.com/bparees
/cc @kubernetes/goog-control-plane
https://github.com/orgs/kubernetes/teams/goog-control-plane
@kubernetes/goog-node https://github.com/orgs/kubernetes/teams/goog-node

—
Reply to this email directly or view it on GitHub
#14561.

[derekwaynecarr]

@smarterclayton

What credentials are used for the push?

I assume this would work similar to ImagePullSecrets, we would have a ImagePushSecret required or something similar if you ran with this hook.

Is this generalizable to commit and not push?

I want to prevent the user from pushing an image from the node that was not the output of their own commit. I am not sure I see a use case for commit no-push. Did you have one in mind?

Is this specific to a certain container runtime?

I thought it was timely to raise this given the discussion on runtime client/server abstraction #13768

Near term, this would be Docker runtime, but I can see the desire of supporting pushing to an arbitrary other endpoint for alternative image formats. At minimum, I think this would be introducing the concept of (1) commit and (2) push to the runtime, but I am not sure we need a hook to the end-user that keeps them separate.

Is this a pod lifecycle behavior that could potentially be described in extensible terms?

I imagined this as similar to RestartPolicy, so like a CompletionPolicy.

Also, doesn't this assume that the container may need to be setup with inputs by another container first?

Good question, @csrwng care to weigh in?

[bparees]

Is there a separate issue that covers being able to commit/push a running pod (ie not just when it exits)? There is some desire to be able to modify a running container and then commit/push the change so you could, for example, make a debug change and then scale up the deployment to test it in a clustered mode, without having to start from scratch with building a new image.

if not, can we discuss including that feature here?

@rcernich fyi

[derekwaynecarr]

@bparees - if you had an API to drive the commit and push, would you still want the hook to do it on container exit?

[bparees]

@derekwaynecarr I think it depends how the api was secured. for builds we'd probably want to call the api via the user's project's build service account. As long as we're ok giving that service account permission to invoke the commit/push api, that's probably the path we'd take over the post-hook approach.

assuming also that the api allowed the commit/push of a pod that had already exited (but not been pruned, obviously)

[bparees]

(the hook approach does offer the advantage of not having a timing window in which the pod gets purged after exiting and before we have a chance to invoke the commit/push api)

[csrwng]

Also, doesn't this assume that the container may need to be setup with inputs by another container first?
Good question, @csrwng care to weigh in?

I would think we could do something similar to what @smarterclayton is proposing with openshift/origin#4819. Another possibility would be to have an intermediary store for the source (internal gitserver) that can be mounted as a volume on the new container.

[ironcladlou]

Just wanted to share a practical downstream use case for this: today, builds in OpenShift produce a defined image output, but since the build pod is responsible for pushing, it's difficult to make the push resilient to failure. Very often, the pod will exit prior to pushing (deadline on the pod, registry timeout, etc) rendering the built image inaccessible. A new build is then required.

Having platform managed pushes seems to solve this neatly.

[smarterclayton]

As a counter design to making this part of the pod API, can we instead
define a way for a separate process/plugin to interact with the lifecycle
of a pod outside of the Kubelet process?

[derekwaynecarr]

Possibly, but how would that outside process know what actions to perform
on what container? We would still need something to introspect on the
pod/container to know to perform the action. Were you thinking the external
process would speak back to the master to know if the pod was associated
with a build? I guess that could work. Needs some more design thought.

On Thursday, October 1, 2015, Clayton Coleman notifications@github.com
wrote:

As a counter design to making this part of the pod API, can we instead
define a way for a separate process/plugin to interact with the lifecycle
of a pod outside of the Kubelet process?

On Oct 1, 2015, at 10:05 PM, Dan Mace <notifications@github.com
javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

Just wanted to share a practical downstream use case for this: today,
builds in OpenShift produce a defined image output, but since the build pod
is responsible for pushing, it's difficult to make the push resilient to
failure. Very often, the pod will exit prior to pushing (deadline on the
pod, registry timeout, etc) rendering the built image inaccessible. A new
build is then required.

Having platform managed pushes seems to solve this neatly.

—
Reply to this email directly or view it on GitHub
<
#14561 (comment)

.

—
Reply to this email directly or view it on GitHub
#14561 (comment)
.

[smarterclayton]

Or it's attached to the Kubelet via an observation mechanism. Committing
the runtime state of a pod isn't fundamental yet, although live migration
and freeze are related. Push is definitely not generic yet. Security on
push is even less generic.

[bgrant0607]

Related to #140

[mqliang]

@bparees I send a issue/pr to that covers being able to commit/push a running pod. see #18205 and #19107 So that user could commit/push their container manually.

[mqliang]

BTW, I think a PostStop hook is also very helpful, since it offer the advantage of not having a timing window in which the pod gets purged after exiting and before we have a chance to invoke the commit/push.

[fejta-bot]

Issues go stale after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Freeze the issue for 90d with /lifecycle frozen.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close