Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support multiple container engines on the host #154

Closed
BenTheElder opened this issue Dec 6, 2018 · 32 comments
Closed

support multiple container engines on the host #154

BenTheElder opened this issue Dec 6, 2018 · 32 comments
Assignees
Labels
kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. kind/feature Categorizes issue or PR as related to a new feature. lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.
Milestone

Comments

@BenTheElder
Copy link
Member

BenTheElder commented Dec 6, 2018

#151 should bring initial support for podman via @rhatdan

This should mostly be a drop in replacement. (it's not... podman has quirks)

We should further clean up the code base a bit to help with this sort of thing. We've mostly but not quite isolated the container runtime functionality.

/assign
/assign @rhatdan

/kind feature
/kind cleanup
/priority important-longterm

related: #153 is the same thing, but within the nodes

@k8s-ci-robot k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. labels Dec 6, 2018
@BenTheElder BenTheElder added this to the 2019 goals milestone Dec 8, 2018
@BenTheElder
Copy link
Member Author

There's now a container package with docker as a subset, and we're working on plumbing through optional arguments to the cluster better, I expect to get limited podman support based on these pretty soon.

@BenTheElder
Copy link
Member Author

We're also targeting CRI functionality explicitly: https://kind.sigs.k8s.io/docs/design/principles/#target-cri-functionality

@BenTheElder
Copy link
Member Author

Currently working on moving more of the flags (ports) behind abstractions borrowed from CRI.
I'd like to isolate all knowledge of docker / podman style flags to pkg/container/... by the next milestone, at which point we should be able to do podman support properly with it's own implementation of all functionality.

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 16, 2019
@tao12345666333
Copy link
Member

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 16, 2019
@tinyzimmer
Copy link

This is pretty old, but in case anyone is still paying attention - I'm pretty close to having this working where basically the global --podman switch uses podman instead.

I had to do something along the lines of what you are describing first, @BenTheElder, where I made a generic Container interface and different backends (e.g. docker,podman) just need to implement the various methods.

package runtime

// .. import pkg/container/backends ...

type ContainerRuntime string

const Podman ContainerRuntime = "podman"
const Docker ContainerRuntime = "docker"

var currentRuntime ContainerRuntime

func SetContainerRuntime(rt ContainerRuntime) {
	currentRuntime = rt
}

func GetContainerRuntime() container.Container {
	switch currentRuntime {
	case Docker:
		return docker.New()
	case Podman:
		return podman.New()
	default:
		panic("invalid container runtime")
	}
}

Then replaced all the places that called the docker package directly with basically a runtime.GetContainerRuntime()

@BenTheElder
Copy link
Member Author

Yeah, sorry this has been drawn out. We're definitely doing this, things have settled a bit and now that I've landed some new debugging infrastructure and other major cleanups I'll be reviving this ...

That said: If you're just replacing docker with podman in all the commands that should be possible today without any changes to the binary:

TMP_DIR=$(mktemp -d)
ln -s "$(which podman)" "${TMP_DIR}/docker"
export PATH="${TMP_DIR}:${PATH}"
kind create cluster

We'd like to do something a little more comprehensive to make things more flexible internally and go beyond even just docker and podman eventually (eg kata or ignite might be quite handy for some use cases) ...

@tinyzimmer
Copy link

tinyzimmer commented Sep 11, 2019

I tried that at first and I couldn't get it to work actually, linux and wsl2 (which is its own ball game entirely). And I might have spoken too soon on my previous comment, as well. Podman...has its quirks still.

So far it was just simple things like format templates being a bit different, but then I ran into an issue with the node failing to start because it couldn't remount /sys - no matter what flags I tried. Running with sudo works, but hopefully I can figure out the real cause of that.

Now for some reason node.WriteFile() when used with podman is not returning an error, but also not writing a file...

This might be an adventure still. But in the spirit of doing something comprehensive, I do at least have it closer in that direction and it doesn't break any of the docker capabilities. It's kind a dud of a branch though if I can't even get a cluster built with podman 😛.

EDIT: I think the next logical step in trying to accomplish this would be to swap the CLI calls with native API calls. github.com/docker/client, github.com/containers/libpod, etc...

There is a regression in podman > 1.5.0 with passing stdin to the container during an exec. I'm getting further with 1.4.4 though.

@BenTheElder
Copy link
Member Author

I have a pretty drastic refactor to put node creation / deletion / interaction behind interfaces ongoing in this branch: https://github.com/BenTheElder/kind/tree/nodes

@BenTheElder BenTheElder added the lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. label Sep 27, 2019
@BenTheElder
Copy link
Member Author

PR for that branch #879

@BenTheElder
Copy link
Member Author

#879 is in, we're most of the way to having this abstracted out for nodes, there's one or two code paths that still depend on specifically docker. I'm working on a follow up to eliminate the rest of those.

@BenTheElder
Copy link
Member Author

#1023 is the last change to remove docker from the critical path for bringing up a cluster and move it behind the provider interface.

I intend to do further cleanup to the APIs around this and experiment with podman etc. in the v0.7 timeframe (AKA O(soon)). We're almost done with the fixes for 0.6

@BenTheElder
Copy link
Member Author

BenTheElder commented Oct 30, 2019

#1023 landed, #1025 finishes removing docker specific things from the APIs, leaving room to start experimenting with Provider implementations.

I've also fired up a throwaway GCE VM with podman for a while now, doing some experiments with that to get a better understanding of what supporting podman properly would entail now.

@rhatdan
Copy link

rhatdan commented Oct 30, 2019

Awesome.

@christianh814
Copy link

Since Fedora 31 basically broke docker for me, I'm just adding this comment so I can follow this. I've been using podman for everything EXCEPT KinD. So this will be great to only use one tool.

@rhatdan - Is there a corresponding being tracked at the podman repo?

As soon as there's things to test, I'll be ready! :)

@BenTheElder
Copy link
Member Author

Since Fedora 31 basically broke docker for me, I'm just adding this comment so I can follow this. I've been using podman for everything EXCEPT KinD. So this will be great to only use one tool.

So, Fedora 31 broke kubernetes. Podman will not fix this. You need to switch to cgroups v1, which will unbreak docker and Kubernetes.

@BenTheElder
Copy link
Member Author

In theory it should be possible to test podman compatibility with kind by doing:

TMP_DIR=$(mktemp -d)
ln -s "$(which podman)" "${TMP_DIR}/docker"
export PATH="${TMP_DIR}:${PATH}"
kind create cluster

With zero kind changes.

But in practice you'll likely find that podman is still not actually 100% compatible to the docker CLI, so we'll need to start implementing a custom backend in kind to handle these.

I have a GCE vm with podman starting on this but it is NOT the priority of this project. The focus is testing Kubernetes for the Kubernetes project, which we are accomplishing with docker for the moment. https://kind.sigs.k8s.io/docs/contributing/project-scope/

@christianh814
Copy link

Since Fedora 31 basically broke docker for me, I'm just adding this comment so I can follow this. I've been using podman for everything EXCEPT KinD. So this will be great to only use one tool.

So, Fedora 31 broke kubernetes. Podman will not fix this. You need to switch to cgroups v1, which will unbreak docker and Kubernetes.

ACK. I will be passing the kernel boot options for now.

@tinyzimmer
Copy link

tinyzimmer commented Nov 16, 2019

I am like 90% done with a podman provider with @BenTheElder 's refactor.

The aforementioned rootless issue still applies. I can't get the kind image to boot properly unless podman (and therefore kind) is executed as root. This also causes the kubeconfig to not get merged properly if you are wanting a non-root user to use kubectl. If someone is wanting to take a swing at this, make sure you first allow your user a crap-load more uids than one might normally would.

E.g. my /etc/sub{u,g}id is

devbox:10000:344930

just to get the container created in rootless podman.

The issue I'm trying to figure out right now is with cleanup. Specifically, the pods in kube-system are somehow persisting across creates and deletes of clusters and I'm not sure why at the moment. This could just be something I'm not grasping about podman, but figured I'd drop in here if any one knows what might cause something like that.

@BenTheElder
Copy link
Member Author

/assign @amwat
We just ~paired on this for a bit, we have something running but the above issues are still true and need verified workarounds :-)

Expect PRs related to this issue soon.

The issue I'm trying to figure out right now is with cleanup. Specifically, the pods in kube-system are somehow persisting across creates and deletes of clusters and I'm not sure why at the moment. This could just be something I'm not grasping about podman, but figured I'd drop in here if any one knows what might cause something like that.

That's because podman until October last year didn't support anonymous volumes properly, instead it bind mounted them to the same path on the host (!)

containers/podman#4287

@amwat
Copy link
Contributor

amwat commented Jan 24, 2020

More updates: We'll need to wait for podman 1.8 to get
containers/podman#4818 that fixes a regression in podman exec.
we use exec heavily, e.g.

return n.Command("cp", "/dev/stdin", dest).SetStdin(strings.NewReader(content)).Run()

@BenTheElder
Copy link
Member Author

can confirm that it's workable with 1.7.1-dev though, needs some more work in kind but feasible once containers/podman#4818 lands.

@rhatdan
Copy link

rhatdan commented Jan 31, 2020

It has landed and hopefully will be in podman 1.8.0.

@rhatdan
Copy link

rhatdan commented Jan 31, 2020

BTW Now that Podman is beginning to support the Docker API, it would be interesting to see if KIND could use it.

@BenTheElder
Copy link
Member Author

#1302 -- provisional support is in, it requires a dev build (1.7.0-dev+) until 1.8.0 is out, then we'll probably start requiring 1.8.0. It also requires rootful for now.

@BenTheElder BenTheElder modified the milestones: 2019 goals, v0.8.0 Feb 21, 2020
@BenTheElder
Copy link
Member Author

In v0.8.0 we will be selecting podman or docker backends automatically. The support is a bit rough (needs rootful, which makes some things clunky) but mostly functional.

There will be more details in the release announcement, probably relatively soon. I'd like to fix up a few smaller things in 0.8

@mathieu-benoit
Copy link

Hi @BenTheElder, just curious, any updates about the 0.8.0 release and/or the support of CRI containers runtime (podman)?

@BenTheElder
Copy link
Member Author

There's an issue tracking the 0.8.0 release #1475

Note that the podman support is experimental and has many limitations, I don't recommend it yet and we're not committed to keeping it for sure, certain desirable features seem to be infeasible (like #148)

@mathieu-benoit
Copy link

Good to know, thanks for the clarification.

@BenTheElder
Copy link
Member Author

Just wanted to update that we've all discussed this and are going forward with committing to the podman backend for now.

It's still under tested and secondary, notably host-reboot functionality will likely be lacking for the immediate future, but we have more core contributor commitment to this.

You can see evidence of this in #1611 #1581 #1506 #1567 #1401 (comment) ...

@afbjorklund
Copy link
Contributor

I would like to cooperate (minikube with kind), regarding the podman support.

We have added a "podman" driver to minikube next to the docker driver,
and a (varlink) "podman-env" next to the minikube docker-env command.

Since the "kicbase" we are using is based on kindbase, trying to upstream all

@BenTheElder
Copy link
Member Author

xref: #1639, thanks for the PR! cc @aojea @amwat ^^

stg-0 added a commit to stg-0/kind that referenced this issue Jun 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. kind/feature Categorizes issue or PR as related to a new feature. lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.
Projects
None yet
Development

No branches or pull requests

10 participants