New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Containerize exec #13138

Closed
rootfs opened this Issue Aug 25, 2015 · 10 comments

Comments

Projects
None yet
4 participants
@rootfs
Copy link
Member

rootfs commented Aug 25, 2015

Containerized exec

exec interface invokes executables that are installed on the host. Containerized exec aims to augument the existing exec interface so Kubelet can execute commands that are not only installed on host but also available on containers, so storage plugins can configure runtime environment without depending on the package availability on the host.

Containerized exec reduces complexities in deploying, configuring, and upgrading K8s host.

Use Cases

Volume Plugins

  • rbd volume plugin executes rbd command that is provided by a Ceph client container.
  • iSCSI volume plugin executes iscsiadmin command to configure iSCSI devices.
  • Fibre Channel volume plugin (WIP) executes commands in sg3-util package to configure Fibre Channel devices.

Scope and limitation

  • Containerized exec does not resolve Linux kernel dependency issue. Containerized exec do not load kernel modules or manage sysfs on the host.
  • Containerized exec is not expected to run indefintely, i.e. not daemonized.
  • Containers that containerize executables are not scheduled by api server.

Structure

// Containerized exec Interface is an interface that presents a subset of the os/exec API.  Use this
// when you want to inject fakeable/mockable exec behavior.
type Interface interface {
    // Command returns a Cmd instance which can be used to run a single command.
    // This follows the pattern of package os/exec.
    // if container is nil, fall back to normal exec
    Command(container *api.Pod, cmd string, args ...string) Cmd
}

Then Command will invoke docker run container_name cmd args

Providing Containerized exec

Pods that provide containerized exec can be hard-coded, user-provided, or namespace scoped configured.

  • hard-coded. Containerized exec callers explicitly specify the name, command, and environment of the container.
  • user-provided. Pod authors provide the container information, along with other clauses in the Pod.
  • namespace scoped. A new k8s resource namespaced k/v service is provided to map utility containers and their intended uses. It is similar to Docker registry or yum repository service. Kubelet uses a reference key (e.g. "rbd client container") and looks up to find the matching container.

cc @kubernetes/rh-cluster-infra @kubernetes/rh-storage

@ncdc

This comment has been minimized.

Copy link
Member

ncdc commented Aug 25, 2015

@rootfs we already support running commands in containers. As a remote client you can run kubectl exec, which ultimately uses the Kubelet to exec the desired command in a container. Is this different from that?

@rootfs

This comment has been minimized.

Copy link
Member

rootfs commented Aug 25, 2015

@ncdc There is a distinction. kubectl exec is cli initiated by apiserver, while containerized exec is invoked as function calls inside kubelet. The idea is to use containers to wrap various executables such as /sbin/mount.nfs, rbd, iscsiadmin, and safe_format_and_mount, etc.

The implementation of this containerized exec will certainly borrow from kubectl exec.

Thanks

@ncdc

This comment has been minimized.

Copy link
Member

ncdc commented Aug 25, 2015

@rootfs ok, that's where I assumed you were headed. Can you give an example of how/where you'd use this? We already have

func (kl *Kubelet) ExecInContainer(podFullName string, podUID types.UID, containerName string, cmd []string, stdin io.Reader, stdout, stderr io.WriteCloser, tty bool) error

so I'm curious what sort of changes are necessary (if any) to support your use cases.

@rootfs

This comment has been minimized.

Copy link
Member

rootfs commented Aug 25, 2015

@ncdc yes, this interface will be heavily and (thus shameless) borrowed.

My understanding is

  • when invoking a container to execute a command, kubelet doesn't have the PodUID yet.
  • the command is expect to return eventually (not long running), so there is timeout mechanism has to be put into place.
@ncdc

This comment has been minimized.

Copy link
Member

ncdc commented Aug 25, 2015

I'm pretty sure that PodUID isn't required for exec to work.

Note: there's unfortunately no way using the default exec handler (which uses the docker exec API) to forcibly kill an exec'd process. I'm not sure how you'd add in a timeout mechanism easily.

@rootfs

This comment has been minimized.

Copy link
Member

rootfs commented Aug 25, 2015

from my poor code reading, the container is still expected to be (already) running so kubelet can find it and use docker exec to execute the command, while in containerized exec, the container is invoked from images via docker run command.

the timeout mechanism doesn't have to be a docker exec option, one rough approach is to use timeout(1)

docker run container timeout 30 some_command

@ncdc

This comment has been minimized.

Copy link
Member

ncdc commented Aug 25, 2015

Ohhhh are you saying you want to do the equivalent of docker run?

@rootfs

This comment has been minimized.

Copy link
Member

rootfs commented Aug 25, 2015

right, should have been more explicit :)

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Aug 27, 2015

I have no problem with this idea, looking forward to the details

On Tue, Aug 25, 2015 at 7:14 AM, Huamin Chen notifications@github.com
wrote:

Containerized exec

exec interface
https://github.com/kubernetes/kubernetes/tree/master/pkg/util/exec
invokes executables that are installed on the host. Containerized exec aims
to augument the existing exec interface so Kubelet can execute commands
that are not only installed on host but also available on containers, so
storage plugins can configure runtime environment without depending on the
package availability on the host.

Containerized exec reduces complexities in deploying, configuring, and
upgrading K8s host.
Use Cases Volume Plugins

  • rbd volume plugin executes rbd command that is provided by a Ceph
    client container.
  • iSCSI volume plugin executes iscsiadmin command to configure iSCSI
    devices.
  • Fibre Channel volume plugin (WIP) executes commands in sg3-util
    package to configure Fibre Channel devices.

Scope and limitation

  • Containerized exec does not resolve Linux kernel dependency issue.
    Containerized exec do not load kernel modules or manage sysfs on the host.
  • Containerized exec is not expected to run indefintely, i.e. not
    daemonized.
  • Containers that containerize executables are not scheduled by api
    server.

Structure

// Containerized exec Interface is an interface that presents a subset of the os/exec API. Use this// when you want to inject fakeable/mockable exec behavior.type Interface interface {
// Command returns a Cmd instance which can be used to run a single command.
// This follows the pattern of package os/exec.
// if container is nil, fall back to normal exec
Command(container *api.Pod, cmd string, args ...string) Cmd
}

Providing Containerized exec

Pods that provide containerized exec can be hard-coded, user-provided, or
namespace scoped configured.

  • hard-coded. Containerized exec callers explicitly specify the name,
    command, and environment of the container.
  • user-provided. Pod authors provide the container information, along
    with other clauses in the Pod.
  • namespace scoped. A new k8s resource namespaced k/v service is
    provided to map utility containers and their intended uses. It is similar
    to Docker registry or yum repository service. Kubelet uses a reference key
    (e.g. "rbd client container") and looks up to find the matching container.


Reply to this email directly or view it on GitHub
#13138.

@rootfs

This comment has been minimized.

Copy link
Member

rootfs commented Sep 21, 2015

replaced by #14288

@rootfs rootfs closed this Sep 21, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment