Add buildah/podman for managing images

[rcarmo]

Is your feature request related to a problem? Please describe.
I require the ability to re-tag and manage images (including pushing to a private registry) inside the k3s environment (typically at the master or single node).

Describe the solution you'd like
I would like to have either buildah or podman baked in, similarly to how crictl is currently made available.

Describe alternatives you've considered
Installing either separately.

Additional context
N/A

[hakuno]

Any news?

[ibuildthecloud]

I'm very interested in this. Basically today k3s does not address the full flow of container development, namely building images. I think it may be companion project too. If I have some time I'll look into this. I would say that I wouldn't expect this soon because the effort is probably substantial. It would be significantly easier if podman, buildah was built on the same libraries as containerd, but they aren't.

[Ciantic]

I'm interested on this because I would like to create k3s images which run e.g. full istio and knative inside a container. With dind (and kind) this is a bit cumbersome.

You may ask why would I want to run istio with k3s in container? For easy and reproducible learning setup.

Podman is great for many reasons, one is that you can pull during a creation of a image, e.g. in Dockerfile

RUN podman pull nginx

Same thing can't be done with dind because it requires dockerd to be running.

[rcarmo]

For me this I see entirely about image management in the cluster itself. When setting up a test/dev cluster, I find myself wanting to create containers on the more often than not, because I just don’t have hat many machines (or the right kind of machines) lying around.

[sandys]

@ibuildthecloud we are now very interested in this . Docker is pretty much not moving on cgroups v2 opencontainers/runc#654

in order to force this issue, Fedora has made cgroups v2 as default and mandatory in the new upcoming Fedora 31 causing docker to fail to run. docker/for-linux#665

Podman (and other docker equivalents) have supported cgroups v2 for years.

Being tied to docker is probably not the best idea at this point.

[sandys]

also related to #900

[AkihiroSuda]

Rio already integrates image builder (buildkit) into k3s

[AkihiroSuda]

@sandys

Anyway Kubernetes doesn't support cgroup2 yet

[justincormack]

@sandys "Docker is pretty much not moving on cgroups v2" is totally false. The runc project is working towards getting cgroups v2 support working; this is a community project with many groups involved, including Docker, Suse and others. You can use Docker with crun which should probably work at this point for v2, although I have not tested this yet.

[AkihiroSuda]

@justincormack

Docker+crun+cgroup2 still doesn't work yet because shim doesn't support cgroup2.

PR: containerd/cgroups#102

[sandys]

Hi Justin My comment was not intended as a blame, but simply highlighting status quo. With the next Fedora drop, there is a high likelihood that we will be locked out of docker. And in the context of os vs docker...OS wins (because of overall security, etc etc etc). Sorry if the words sounded unfriendly

…

On Mon, 28 Oct, 2019, 17:15 Akihiro Suda, ***@***.***> wrote: @justincormack <https://github.com/justincormack> Docker+crun+cgroup2 still doesn't work yet because shim doesn't support cgroup2. PR: containerd/cgroups#102 <containerd/cgroups#102> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#488>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAASYUZQT6VTZ5S45AO2IKDQQ3GFRANCNFSM4HN3EC3A> .

[rcarmo]

I’m just going to add a little note here that I’d be perfectly OK with something else but podman (in fact, in the months since I opened this issue, I’ve had a few issues with podman and would rather not rely on it).

[leigh-j]

@sandys

Anyway Kubernetes doesn't support cgroup2 yet

containerd and runc dont support cgroup2 yet, there are alternatives. I believe this issue is best for tracking containerd catching up.
containerd/cgroups#104

[AkihiroSuda]

containerd and runc dont support cgroup2 yet

Both supports cgroup2

[MaciejKucia]

Well it seems to be a problem for me:

Apr 13 12:33:56 maciej.workbook k3s[182605]: time="2020-04-13T12:33:56.388735776+08:00" level=info msg="k3s is up and running"
Apr 13 12:33:56 maciej.workbook k3s[182605]: time="2020-04-13T12:33:56.388832649+08:00" level=warning msg="Failed to find cpuset cgroup, you may need to add \"cgroup_enable=cpuset\" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)"
Apr 13 12:33:56 maciej.workbook k3s[182605]: time="2020-04-13T12:33:56.388849130+08:00" level=error msg="Failed to find memory cgroup, you may need to add \"cgroup_memory=1 cgroup_enable=memory\" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)"
Apr 13 12:33:56 maciej.workbook k3s[182605]: time="2020-04-13T12:33:56.388865737+08:00" level=fatal msg="failed to find memory cgroup, you may need to add \"cgroup_memory=1 cgroup_enable=memory\" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)"

$ uname -a
Linux maciej.workbook 5.5.15-200.fc31.x86_64 #1 SMP Thu Apr 2 19:16:17 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

$ k3s --version
k3s version v1.17.4+k3s1 (3eee8ac3)

[johndiego]

news solutions ?

[vitas]

also interewsting in that

[olinfc]

Following thread.

Per PR #2584:

Support cgroup v2.

This PR does not add support for rootless cgroup yet. Rootless cgroup will be supported in a separate PR soon.

[brandond]

FWIW we will probably be filling this gap by integrating https://github.com/rancher/k3c

[rcarmo]

Well, as long as there is a simple option for having a private registry, possibly even an in-cluster one that can work without Internet access - possibly also without TLS, since it is a major pain for IoT and industrial environments without connectivity - I'm good.

[cjellick]

@dweomer assigning to you. Can you add or link to your design of k3c/k3b here?

[AkihiroSuda]

W.r.t. the original discussion about images, nerdctl can be used for managing images in the containerd store: https://github.com/AkihiroSuda/nerdctl

e.g. nerdctl --namespace=k8s.io load <OCI>, nerdctl --namespace=k8s.io tag <OLD> <NEW>, nerdctl --namespace=k8s.io push <IMAGE>

[bbros-dev]

From our PoV k3c seems a distraction.
It would be sufficient if k3s checked the locations Podman stored images and pulled them from there. Of course it is very likely that, upon looking into the implementation, it'd be easier to call out to Podman or Buildah libraries or cli (when they are installed)?

This can be done as the last resort before flagging a not found error - that might be enough to preserve existing behavior?

[trallnag]

@bbros-dev, bit unrelated to this issue: Can I install podman next to k3s? Or will this break things?

[bbros-dev]

@bbros-dev, bit unrelated to this issue: Can I install podman next to k3s? Or will this break things?

We aren't k3s-io devs.... but our understanding is that k3c duplicates the Podman/Buildah/Skopeo functionality, so you should be able to run them side by side.

To be safe maybe try running Buildah along side k3s? Right now, as best we can tell (and from week old memory), you cannot access Podman managed images from k3s.

[rcarmo]

So this is closed without a definitive answer?

[trallnag]

@rcarmo, from what I understand the answer is to use nerdctl for managing images and a separate stack of podman / buildah and so on for building images