New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run kaniko as non-root user within the container #105

Open
priyawadhwa opened this Issue Apr 17, 2018 · 18 comments

Comments

Projects
None yet
7 participants
@priyawadhwa
Copy link
Member

priyawadhwa commented Apr 17, 2018

No description provided.

@dlorenc

This comment has been minimized.

Copy link
Member

dlorenc commented Apr 17, 2018

Right now kaniko requires root for a few reasons:

  • To unpack the base image into / with proper permissions
  • To execute RUN commands that require root

We should be able to solve the first case using an init container. This container will copy the executor binary into a shared volume.

Then the "build" container image is actually the base image in the FROM line, and the entrypoint gets set to the binary we just copied into the shared volume.

This also takes advantage of layer caching by letting the underlying runtime unpack the base layers.

We can't really fix the second part until something like user namespaces are supported in kubernetes, but we can allow builds to run with only the permissions they need until then.

@cyphar

This comment has been minimized.

Copy link
Contributor

cyphar commented Apr 17, 2018

Using the tools we've been developing as part of the rootless-containers project (https://github.com/rootless-containers) it is possible to do both of those things completely unprivileged (though it does require ptrace to do it convincingly). You can even do it inside a container, by nesting the rootless container inside the outer one (though at the moment there are some runc bugs that we are working on resolving).

@dlorenc

This comment has been minimized.

Copy link
Member

dlorenc commented Apr 17, 2018

Yeah, the ptrace trick is something we've looked at with fakeroot-ng (although ptrace is blocked by the default docker capability set). Do you have links to the runc bugs you're working through?

@dlorenc

This comment has been minimized.

Copy link
Member

dlorenc commented Apr 17, 2018

@cyphar do you have any links on running runc inside a container without something like the "--privileged" flag?

@AkihiroSuda

This comment has been minimized.

Copy link

AkihiroSuda commented Apr 18, 2018

@dlorenc

There are two options:

1. Modify container runtime implementations to support RawAccess-ing /proc

Docker-side PR: moby/moby#36644
Kubernetes-side proposal: kubernetes/community#1934
Jess's blog: https://blog.jessfraz.com/post/building-container-images-securely-on-kubernetes/

2. Modify kernel to allow mounting unprivileged and non-fully visible procfs

https://patchwork.kernel.org/patch/10322481/

@cyphar

This comment has been minimized.

Copy link
Contributor

cyphar commented Apr 18, 2018

Yeah, the ptrace trick is something we've looked at with fakeroot-ng (although ptrace is blocked by the default docker capability set).

I would have to double-check this, but when you set up a rootless container you have the full capability set (though we drop capabilities just as in the privileged case, so you'd need to add CAP_SYS_PTRACE to the capability set). So it is possible that the seccomp profile will not blovk ptrace once inside the rootless container (I will have to test this first though, since I'm not sure which *_capable variant is being used by seccomp in that context.

do you have any links on running runc inside a container without something like the "--privileged" flag?

At the moment the main issue us the /proc mounting problem that @AkihiroSuda just mentioned. Aside from that issue, you should be able to just run a rootless container in a Docker container (with the note that you need to whitelist unshare(CLONE_NEWUSER) in the seccomp profile). Another fix for the /proc mounting issue is to mount an empty procfs from an empty pid namespace inside the Docker container, which will fix the EPERM you currently get.

@r2d4

This comment has been minimized.

Copy link
Member

r2d4 commented Apr 18, 2018

Why do you need a nested runtime to begin with?

Runtime configuration security can always be configured in the "parent" build container, seccomp profiles applied, etc. You would do this in Kubernetes by setting pod security policies (of course not all the features are fully implemented today).

Kaniko is meant to be ran from inside a container only and never as a standalone "runtime". It almost seems like the wrong separation of concerns to have set of default security applied during the "parent" container, and then have the nested container apply something else (albeit only more strict).

I can't imagine nested containers would be easier to manage with Kubernetes non-container security policies like RBAC, since you would really like to scope whoever is running the build to the nested non-root container policies, which is the minimal scope.

@cyphar

This comment has been minimized.

Copy link
Contributor

cyphar commented Apr 19, 2018

The nested containers suggestion is so that the build doesn't require root (which it currently does). Docker (or cri-o or Kubernetes -- though we are working on those slowly but surely) doesn't support rootless containers.

@r2d4

This comment has been minimized.

Copy link
Member

r2d4 commented Apr 19, 2018

Ah I guess I meant once docker or cri-o supports rootless containers, then we won't need the nested container? The only purpose of the nested container is that the parent runtime doesnt support rootless while the nested runtime does.

@cyphar

This comment has been minimized.

Copy link
Contributor

cyphar commented Apr 20, 2018

Sure, though with Docker and cri-o you would still be in a bit of grey area (in most cases I would assume the kubelet would be running as root even if we had rootless support in the entire stack -- though the CloudFoundry folks have shown that people are actually okay with rootless in a lot of circumstances).

Really it depends how much you want to push the rootless thing. Personally (for the stuff I'm working on) I think that if you have any code-path that requires uid=0 then it is not an acceptable solution, so I wouldn't classify having cri-o (as root) spawning "rootless" containers as being acceptable. But of course different people have different levels of fanaticism. 😸

(Note that Docker will likely never support rootless directly, purely due to the amount of work required -- though I'd be happy to be proven wrong. containerd has some patches for it but it requires running in a specific environment unlike umoci and similar.)

@ianmiell

This comment has been minimized.

Copy link

ianmiell commented Apr 20, 2018

This is a very disappointing thread to read. I was given to understand from the publicity that kaniko did not require any privileges:

https://cloudplatform.googleblog.com/2018/04/introducing-kaniko-Build-container-images-in-Kubernetes-and-Google-Container-Builder-even-without-root-access.html

this is a bit of a holy grail for our org, since the build environment is very locked-down.

@cyphar

This comment has been minimized.

Copy link
Contributor

cyphar commented Apr 20, 2018

@ianmiell You can do unprivileged builds with umoci. If you prefer Dockerfile-style builds there's orca-build (I am currently working on combining orca-build into the umoci project so it's more obvious). There's also a lot of rootless containers work being done in https://github.com/rootless-containers.

Several organisations already use umoci because it provides completely unprivileged builds (you can pair with rootless runc to also run containers with it). See https://rootlesscontaine.rs/ for some more detail of what is going on there.

@ianmiell

This comment has been minimized.

Copy link

ianmiell commented Apr 20, 2018

@cyphar thanks, but I was hoping for something more ready-packaged. I've a house full of yaks already :)

@dlorenc

This comment has been minimized.

Copy link
Member

dlorenc commented Apr 21, 2018

I agree we need to clarify the documentation and messaging a bit more here.

Kaniko was designed to run in a container in any kubernetes cluster, today. umoci and a few other tools are really cool too, kubernetes and docker just don't support the necessary settings to let them run without the --privileged flag.

@ianmiell, can you describe exactly how your build environment is locked down today?

This issue is about letting kaniko run without root at all, provided your build doesn't require root to run. User namespaces will be required to let builds that require root run without root on the host, but that's not possible today in kubernetes without the --privileged flag or something similar.

@ianmiell

This comment has been minimized.

Copy link

ianmiell commented Apr 22, 2018

If anyone's interested, I've got a reproducible build of a poc using the tools described by @cyphar above (and with his help) here:

https://github.com/ianmiell/shutit-orca-build

on Centos. Does require a little sysctl work to enable (in the script).

@cyphar

This comment has been minimized.

Copy link
Contributor

cyphar commented Apr 22, 2018

Regarding yak-shaving, yeah we're working on it. @AkihiroSuda has runrootless which wraps a lot of the stuff we have been working on -- but at the moment there are quite a few other problems that need to be solved before we can even start polishing it (unprivileged network bridges for instance, as well as a lot of other ancillary kernel work such as bind-mount mappings). And the current ptrace method that PRoot uses will no longer be necessary once @tych0 gets full-on seccomp syscall emulation merged into Linux (making emulation much faster, and easier to do (in theory)). A lot of this stuff is still changing.

@norpol

This comment has been minimized.

Copy link

norpol commented Sep 25, 2018

Just stumbled upon Kaniko once more and asking myself if with the Docker gVisor support, is it now possible to build images without root or docker-in-docker within a Docker or Kubernetes host?

@dlorenc

This comment has been minimized.

Copy link
Member

dlorenc commented Sep 25, 2018

Yup, if you have gvisor configured you should be good to go: https://github.com/GoogleContainerTools/kaniko/blob/master/README.md#running-kaniko-in-gvisor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment