Support the user flag from docker exec in kubectl exec

[VikParuchuri]

It looks like docker exec is being used as the backend for kubectl exec. docker exec has the --user flag, which allows you to run a command as a particular user. This same functionality doesn't exist in Kubernetes.

Our use case is that we spin up pods, and execute untrusted code in them. However, there are times when after creating the pod, we need to run programs that need root access (they need to access privileged ports, etc).

We don't want to run the untrusted code as root in the container, which prevents us from just escalating permissions for all programs.

I looked around for references to this problem, but only found this StackOverflow answer from last year -- http://stackoverflow.com/questions/33293265/execute-command-into-kubernetes-pod-as-other-user .

There are some workarounds to this, such as setting up a server in the container that takes commands in, or defaulting to root, but dropping to another user before running untrusted code. However, these workarounds break nice Kubernetes/Docker abstractions and introduce security holes.

[adohe]

SGTM. @kubernetes/kubectl any thoughts on this?

[smarterclayton]

It's not unreasonable, but we'd need pod security policy to control the user input and we'd probably have to disallow user by name (since we don't allow it for containers - you must specify UID).

[smarterclayton]

@sttts and @ncdc re exec

[sttts]

Legitimate use-case

[killdash9]

Any update on this?

[miracle2k]

My app container image is built using buildpacks. I'd like to open a shell. When I do, I am root, and all the env vars are set. But the buildpack-generated environment is not there. If I open a login shell for the app user (su -l u22055) I have my app environment, but now the kubernetes env vars are missing.

[smarterclayton]

I thought su -l didn't copy env vars? You have to explicitly do the copy
yourself or use a different command.

On Tue, Oct 11, 2016 at 5:26 PM, Michael Elsdörfer <notifications@github.com

wrote:

My app container image is built using buildpacks. I'd like to open a
shell. When I do, I am root, and all the env vars are set. But the
buildpack-generated environment is not there. If I open a login shell for
the app user (su -l u22055) I have my app environment, but now the
kubernetes env vars are missing.

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
#30656 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_p7sIu20xnja2HsbPUUgD1m4gXqVAks5qzCksgaJpZM4Jk3n0
.

[adarshaj]

@miracle2k - Have you tried su -m -l u22055? -m is supposed to preserve environment variables.

[miracle2k]

@adarshaj @smarterclayton Thanks for the tips. su -m has it's own issues (the home dir is wrong), but I did make it work in the meantime. The point though is - that's why I posted it here - is that I'd like to see "kubectl exec" do the right thing. Maybe even use the user that the docker file defines.

[jsindy]

Here is an example how I need this functionality.

The official Jenkins image runs as the user Jenkins. I have a persistent disk attached that I need to resize. If kubectl had the --user I could bash in as root and resize2fs. Unfortunately without it it is an extreme pain.

[chrishiestand]

An additional use case - you're being security conscious so all processes running inside the container are not privileged. But now something unexpectedly isn't working and you want to go in as root to e.g. install debug utilities and figure out what's wrong on the live system.

[SimenB]

Installing stuff for debugging purposes is my use case as well. Currently I ssh into the nodes running kubernetes, and use docker exec directly.

[gaballard]

What's the status on this? This functionality would be highly useful

[fabianofranz]

I didn't check, but does the --as and --as-group global flags help here? Do they even work with exec? cc @liggitt

[liggitt]

I didn't check, but does the --as and --as-group global flags help here? Do they even work with exec? cc @liggitt

No, those have to do with identifying yourself to the kubernetes API, not passing through to inform the chosen uid for the exec call

[whereisaaron]

The lack of the user flag is a hassle. Use case is I have a container that runs as an unprivileged user, I mount a volume on it, but the volume folder is not owned by the user. There is no option to mount the volume with specified permissions. I can't use an entrypoint script to change the permissions because that runs as the unprivileged user. I can't use a lifecycle.preStart hook because that runs as the unprivileged user too. kubectl exec -u root could do that, if the '-u' option existed.

I guess though this should be an additional RBAC permission, to allow/block 'exec' as other than the container user.

Ideally the lifeCycle hooks should be able to run as root in the container, even when the container does not. Right now the best alternative is probably to run an init container against the same mount; kind of an overhead to start a separate container and mount volumes, when really I just need a one-line command as root at container start.

[xiangpengzhao]

/sig cli

[skorski]

+1 for this feature. Not having this makes debugging things a lot more painful.

[johnjjung]

+1 for this feature. I have to rebuild my docker container and make sure the Docker file has USER root as the last line, then debug, then disable this.

docker command line seems to have a --user flag

[BenAbineriBubble]

johnjjung, if you have ssh access to the node you can connect to the container using docker with the user flag which might save you a bit of time.

[johnjjung]

Hmm, awesome let me try this

…

On Jul 10, 2017, 11:34 -0400, BenAbineriBubble ***@***.***>, wrote: johnjjung, if you have ssh access to the node you can connect to the container using docker with the user flag which might save you a bit of time. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[jiaj12]

+1 really a issue, I have to ssh and then exec the docker exec, such annoying

[sttts]

/cc @frobware

[dims]

/sig node

[Nowaker]

There seems to be a rebase-needing unfinished PR for this feature: #59092. Is there anyone who could pick it up and finalize? CC @louyihua

[alexandersauerbier]

+1
(I run into the same issue as @SimenB - I'd like to install stuff for debugging purposes. Bye the way, thanks for the "use docker directly" hint )

[haizaar]

While we are waiting for this to be properly supported, an intermediate solution can be to run your docker CMD with su-exec (either in Dockerfile or in K8s manifest). su-exec weighs only 20k (on alpine) and this way your app will run unprivileged, while still having root in kubectl exec.

[greenoid]

+1

[sniederm]

I would also appreciate such a -u flag. +1.

Just an idea:

For example something like a --conainer-type would be a big plus to enable passing any supported arguments directly to the underyling container implementation:

kubectl exec --container-type=docker -it -u 0 NAME

This would avoid having only a subset of the underlying functionality of the container runtime in kubectl. Furthermore it saves effort since there is no need to map and abstract the supported arguments from the kubelet layer all the way down to the container for every supported container type.

So in summary without the --container-type flag only the abstracted arguments from kubectl can be used and the underlying container type is completely transparent. With the flag, container specific arguments can be passed along. It's up to the user of kubectl whether he wants to bind to a container type or not.

BTW: Thanks to @SimenB for the hint to ssh into the node and use Docker directly. That solved my problem temporarily. Using Minikube I was able to do the following to log-in as root:
minikube ssh "docker exec -it -u 0 <container-id> bash"
Maybe this could be of help to someone.

[Nowaker]

A workaround script that automates the unpleasant. SSH access to the node required.

Usage:

./shell-into-pod-as-root.sh podname
./shell-into-pod-as-root.sh podname sh

Enjoy!

#!/usr/bin/env bash
set -xe

POD=$(kubectl describe pod "$1")
NODE=$(echo "$POD" | grep -m1 Node | awk -F'/' '{print $2}')
CONTAINER=$(echo "$POD" | grep -m1 'Container ID' | awk -F 'docker://' '{print $2}')

CONTAINER_SHELL=${2:-bash}

set +e

ssh -t "$NODE" sudo docker exec --user 0 -it "$CONTAINER" "$CONTAINER_SHELL"

if [ "$?" -gt 0 ]; then
  set +x
  echo 'SSH into pod failed. If you see an error message similar to "executable file not found in $PATH", please try:'
  echo "$0 $1 sh"
fi

[AndrewSav]

@Nowaker how do you handle namespaces?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[Gregy]

/remove-lifecycle stale

[bryanhuntesl]

I would also appreciate such a -u flag. +1.

Just an idea:

For example something like a --conainer-type would be a big plus to enable passing any supported arguments directly to the underyling container implementation:

kubectl exec --container-type=docker -it -u 0 NAME

This would avoid having only a subset of the underlying functionality of the container runtime in kubectl. Furthermore it saves effort since there is no need to map and abstract the supported arguments from the kubelet layer all the way down to the container for every supported container type.

So in summary without the --container-type flag only the abstracted arguments from kubectl can be used and the underlying container type is completely transparent. With the flag, container specific arguments can be passed along. It's up to the user of kubectl whether he wants to bind to a container type or not.

BTW: Thanks to @SimenB for the hint to ssh into the node and use Docker directly. That solved my problem temporarily. Using Minikube I was able to do the following to log-in as root:
minikube ssh "docker exec -it -u 0 <container-id> bash"
Maybe this could be of help to someone.

Yeah - it's trivial to just use the docker exec to do this - it's mostly about consistency - multi-user docker containers are a bit of a joke really - a legacy from converting a VM to a container.

I'm dealing with this with grafana at the moment - I suppose this will pass with time.

[jordanwilson230]

@bryanhuntesl There's discussion of workarounds above which doesn't require manual ssh'ing to a node. You can also try this plugin -- https://github.com/jordanwilson230/kubectl-plugins

[gjcarneiro]

What if you don't want to allow users to ssh into a node? Allowing users ssh access to a node, as well as allowing them to have access to docker, can be a security risk. Docker doesn't know anything about namespaces or k8s permissions. If a user can run docker exec, it can exec into pods of any namespace.

SSH is not a proper solution, IMHO.

[bryanhuntesl]

What if you don't want to allow users to ssh into a node? Allowing users ssh access to a node, as well as allowing them to have access to docker, can be a security risk. Docker doesn't know anything about namespaces or k8s permissions. If a user can run docker exec, it can exec into pods of any namespace.

SSH is not a proper solution, IMHO.

I second that opinion - using an out of band mechanism to gain direct access is increasing the potential attack area.

[jordanwilson230]

What if you don't want to allow users to ssh into a node? Allowing users ssh access to a node, as well as allowing them to have access to docker, can be a security risk. Docker doesn't know anything about namespaces or k8s permissions. If a user can run docker exec, it can exec into pods of any namespace.

SSH is not a proper solution, IMHO.

There are solutions that do not require SSH @gjcarneiro. Also, a user must first add their public SSH key in the Compute Metadata before they are allowed SSH access to a node (if on GCP) @bryanhuntesl.

[max88991]

@liggitt It's been three years since this topic started, any conclusions?

[davidxia]

I'll write a KEP real soon. Wanted to do it to learn and go through the process and hopefully make a contribution.

[gitnik]

I am not sure if this solution has been mentioned before but what we did as a workaround is have all our containers include a script that'll log you in as the correct user. Plus a motd:

Dockerfile:

USER root
RUN echo "su -s /bin/bash www-data" >> /root/.bashrc
# this exit statement here is needed in order to exit from the new shell directly or else you need to type exit twice
RUN echo "exit" >> /root/.bashrc
# /var/www is www-data's home directory
COPY motd.sh /var/www/.bashrc

motd.sh:

RED='\033[0;31m'
YELLOW='\033[0;33m'

echo -e "${RED}"
echo "##################################################################"
echo "#        You've been automatically logged in as www-data.        #"
echo "##################################################################"
echo -e "${YELLOW} "
echo "If you want to login as root instead:"
echo -e "$(if [ "$KUBERNETES_PORT" ]; then echo 'kubectl'; else echo 'docker'; fi) exec -ti $(hostname) -- bash --noprofile -norc"

TEXT_RESET='\033[0m'

echo -e "${TEXT_RESET} "