Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support multiple container engines on the host #154

Closed
BenTheElder opened this issue Dec 6, 2018 · 32 comments
Closed

support multiple container engines on the host #154

BenTheElder opened this issue Dec 6, 2018 · 32 comments

Comments

@BenTheElder
Copy link
Member

@BenTheElder BenTheElder commented Dec 6, 2018

#151 should bring initial support for podman via @rhatdan

This should mostly be a drop in replacement. (it's not... podman has quirks)

We should further clean up the code base a bit to help with this sort of thing. We've mostly but not quite isolated the container runtime functionality.

/assign
/assign @rhatdan

/kind feature
/kind cleanup
/priority important-longterm

related: #153 is the same thing, but within the nodes

@BenTheElder
Copy link
Member Author

@BenTheElder BenTheElder commented Feb 24, 2019

There's now a container package with docker as a subset, and we're working on plumbing through optional arguments to the cluster better, I expect to get limited podman support based on these pretty soon.

@BenTheElder
Copy link
Member Author

@BenTheElder BenTheElder commented Feb 24, 2019

We're also targeting CRI functionality explicitly: https://kind.sigs.k8s.io/docs/design/principles/#target-cri-functionality

@BenTheElder
Copy link
Member Author

@BenTheElder BenTheElder commented Mar 18, 2019

Currently working on moving more of the flags (ports) behind abstractions borrowed from CRI.
I'd like to isolate all knowledge of docker / podman style flags to pkg/container/... by the next milestone, at which point we should be able to do podman support properly with it's own implementation of all functionality.

@fejta-bot
Copy link

@fejta-bot fejta-bot commented Jun 16, 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@tao12345666333
Copy link
Member

@tao12345666333 tao12345666333 commented Jun 16, 2019

/remove-lifecycle stale

@tinyzimmer
Copy link

@tinyzimmer tinyzimmer commented Sep 11, 2019

This is pretty old, but in case anyone is still paying attention - I'm pretty close to having this working where basically the global --podman switch uses podman instead.

I had to do something along the lines of what you are describing first, @BenTheElder, where I made a generic Container interface and different backends (e.g. docker,podman) just need to implement the various methods.

package runtime

// .. import pkg/container/backends ...

type ContainerRuntime string

const Podman ContainerRuntime = "podman"
const Docker ContainerRuntime = "docker"

var currentRuntime ContainerRuntime

func SetContainerRuntime(rt ContainerRuntime) {
	currentRuntime = rt
}

func GetContainerRuntime() container.Container {
	switch currentRuntime {
	case Docker:
		return docker.New()
	case Podman:
		return podman.New()
	default:
		panic("invalid container runtime")
	}
}

Then replaced all the places that called the docker package directly with basically a runtime.GetContainerRuntime()

@BenTheElder
Copy link
Member Author

@BenTheElder BenTheElder commented Sep 11, 2019

Yeah, sorry this has been drawn out. We're definitely doing this, things have settled a bit and now that I've landed some new debugging infrastructure and other major cleanups I'll be reviving this ...

That said: If you're just replacing docker with podman in all the commands that should be possible today without any changes to the binary:

TMP_DIR=$(mktemp -d)
ln -s "$(which podman)" "${TMP_DIR}/docker"
export PATH="${TMP_DIR}:${PATH}"
kind create cluster

We'd like to do something a little more comprehensive to make things more flexible internally and go beyond even just docker and podman eventually (eg kata or ignite might be quite handy for some use cases) ...

@tinyzimmer
Copy link

@tinyzimmer tinyzimmer commented Sep 11, 2019

I tried that at first and I couldn't get it to work actually, linux and wsl2 (which is its own ball game entirely). And I might have spoken too soon on my previous comment, as well. Podman...has its quirks still.

So far it was just simple things like format templates being a bit different, but then I ran into an issue with the node failing to start because it couldn't remount /sys - no matter what flags I tried. Running with sudo works, but hopefully I can figure out the real cause of that.

Now for some reason node.WriteFile() when used with podman is not returning an error, but also not writing a file...

This might be an adventure still. But in the spirit of doing something comprehensive, I do at least have it closer in that direction and it doesn't break any of the docker capabilities. It's kind a dud of a branch though if I can't even get a cluster built with podman 😛.

EDIT: I think the next logical step in trying to accomplish this would be to swap the CLI calls with native API calls. github.com/docker/client, github.com/containers/libpod, etc...

There is a regression in podman > 1.5.0 with passing stdin to the container during an exec. I'm getting further with 1.4.4 though.

@BenTheElder
Copy link
Member Author

@BenTheElder BenTheElder commented Sep 27, 2019

I have a pretty drastic refactor to put node creation / deletion / interaction behind interfaces ongoing in this branch: https://github.com/BenTheElder/kind/tree/nodes

@BenTheElder
Copy link
Member Author

@BenTheElder BenTheElder commented Sep 27, 2019

PR for that branch #879

@BenTheElder
Copy link
Member Author

@BenTheElder BenTheElder commented Sep 28, 2019

#879 is in, we're most of the way to having this abstracted out for nodes, there's one or two code paths that still depend on specifically docker. I'm working on a follow up to eliminate the rest of those.

@BenTheElder
Copy link
Member Author

@BenTheElder BenTheElder commented Oct 29, 2019

#1023 is the last change to remove docker from the critical path for bringing up a cluster and move it behind the provider interface.

I intend to do further cleanup to the APIs around this and experiment with podman etc. in the v0.7 timeframe (AKA O(soon)). We're almost done with the fixes for 0.6

@BenTheElder
Copy link
Member Author

@BenTheElder BenTheElder commented Oct 30, 2019

#1023 landed, #1025 finishes removing docker specific things from the APIs, leaving room to start experimenting with Provider implementations.

I've also fired up a throwaway GCE VM with podman for a while now, doing some experiments with that to get a better understanding of what supporting podman properly would entail now.

@rhatdan
Copy link

@rhatdan rhatdan commented Oct 30, 2019

Awesome.

@christianh814
Copy link

@christianh814 christianh814 commented Nov 8, 2019

Since Fedora 31 basically broke docker for me, I'm just adding this comment so I can follow this. I've been using podman for everything EXCEPT KinD. So this will be great to only use one tool.

@rhatdan - Is there a corresponding being tracked at the podman repo?

As soon as there's things to test, I'll be ready! :)

@BenTheElder
Copy link
Member Author

@BenTheElder BenTheElder commented Nov 8, 2019

Since Fedora 31 basically broke docker for me, I'm just adding this comment so I can follow this. I've been using podman for everything EXCEPT KinD. So this will be great to only use one tool.

So, Fedora 31 broke kubernetes. Podman will not fix this. You need to switch to cgroups v1, which will unbreak docker and Kubernetes.

@BenTheElder
Copy link
Member Author

@BenTheElder BenTheElder commented Nov 8, 2019

In theory it should be possible to test podman compatibility with kind by doing:

TMP_DIR=$(mktemp -d)
ln -s "$(which podman)" "${TMP_DIR}/docker"
export PATH="${TMP_DIR}:${PATH}"
kind create cluster

With zero kind changes.

But in practice you'll likely find that podman is still not actually 100% compatible to the docker CLI, so we'll need to start implementing a custom backend in kind to handle these.

I have a GCE vm with podman starting on this but it is NOT the priority of this project. The focus is testing Kubernetes for the Kubernetes project, which we are accomplishing with docker for the moment. https://kind.sigs.k8s.io/docs/contributing/project-scope/

@christianh814
Copy link

@christianh814 christianh814 commented Nov 8, 2019

Since Fedora 31 basically broke docker for me, I'm just adding this comment so I can follow this. I've been using podman for everything EXCEPT KinD. So this will be great to only use one tool.

So, Fedora 31 broke kubernetes. Podman will not fix this. You need to switch to cgroups v1, which will unbreak docker and Kubernetes.

ACK. I will be passing the kernel boot options for now.

@tinyzimmer
Copy link

@tinyzimmer tinyzimmer commented Nov 16, 2019

I am like 90% done with a podman provider with @BenTheElder 's refactor.

The aforementioned rootless issue still applies. I can't get the kind image to boot properly unless podman (and therefore kind) is executed as root. This also causes the kubeconfig to not get merged properly if you are wanting a non-root user to use kubectl. If someone is wanting to take a swing at this, make sure you first allow your user a crap-load more uids than one might normally would.

E.g. my /etc/sub{u,g}id is

devbox:10000:344930

just to get the container created in rootless podman.

The issue I'm trying to figure out right now is with cleanup. Specifically, the pods in kube-system are somehow persisting across creates and deletes of clusters and I'm not sure why at the moment. This could just be something I'm not grasping about podman, but figured I'd drop in here if any one knows what might cause something like that.

@BenTheElder
Copy link
Member Author

@BenTheElder BenTheElder commented Jan 23, 2020

/assign @amwat
We just ~paired on this for a bit, we have something running but the above issues are still true and need verified workarounds :-)

Expect PRs related to this issue soon.

The issue I'm trying to figure out right now is with cleanup. Specifically, the pods in kube-system are somehow persisting across creates and deletes of clusters and I'm not sure why at the moment. This could just be something I'm not grasping about podman, but figured I'd drop in here if any one knows what might cause something like that.

That's because podman until October last year didn't support anonymous volumes properly, instead it bind mounted them to the same path on the host (!)

containers/podman#4287

@amwat
Copy link
Contributor

@amwat amwat commented Jan 24, 2020

More updates: We'll need to wait for podman 1.8 to get
containers/podman#4818 that fixes a regression in podman exec.
we use exec heavily, e.g.

return n.Command("cp", "/dev/stdin", dest).SetStdin(strings.NewReader(content)).Run()

@BenTheElder
Copy link
Member Author

@BenTheElder BenTheElder commented Jan 30, 2020

can confirm that it's workable with 1.7.1-dev though, needs some more work in kind but feasible once containers/podman#4818 lands.

@rhatdan
Copy link

@rhatdan rhatdan commented Jan 31, 2020

It has landed and hopefully will be in podman 1.8.0.

@rhatdan
Copy link

@rhatdan rhatdan commented Jan 31, 2020

BTW Now that Podman is beginning to support the Docker API, it would be interesting to see if KIND could use it.

@BenTheElder
Copy link
Member Author

@BenTheElder BenTheElder commented Feb 1, 2020

#1302 -- provisional support is in, it requires a dev build (1.7.0-dev+) until 1.8.0 is out, then we'll probably start requiring 1.8.0. It also requires rootful for now.

@BenTheElder BenTheElder removed this from the 2019 goals milestone Feb 21, 2020
@BenTheElder BenTheElder added this to the v0.8.0 milestone Feb 21, 2020
@BenTheElder
Copy link
Member Author

@BenTheElder BenTheElder commented Feb 21, 2020

In v0.8.0 we will be selecting podman or docker backends automatically. The support is a bit rough (needs rootful, which makes some things clunky) but mostly functional.

There will be more details in the release announcement, probably relatively soon. I'd like to fix up a few smaller things in 0.8

@mathieu-benoit
Copy link

@mathieu-benoit mathieu-benoit commented Apr 29, 2020

Hi @BenTheElder, just curious, any updates about the 0.8.0 release and/or the support of CRI containers runtime (podman)?

@BenTheElder
Copy link
Member Author

@BenTheElder BenTheElder commented Apr 29, 2020

There's an issue tracking the 0.8.0 release #1475

Note that the podman support is experimental and has many limitations, I don't recommend it yet and we're not committed to keeping it for sure, certain desirable features seem to be infeasible (like #148)

@mathieu-benoit
Copy link

@mathieu-benoit mathieu-benoit commented Apr 29, 2020

Good to know, thanks for the clarification.

@BenTheElder
Copy link
Member Author

@BenTheElder BenTheElder commented May 20, 2020

Just wanted to update that we've all discussed this and are going forward with committing to the podman backend for now.

It's still under tested and secondary, notably host-reboot functionality will likely be lacking for the immediate future, but we have more core contributor commitment to this.

You can see evidence of this in #1611 #1581 #1506 #1567 #1401 (comment) ...

@afbjorklund
Copy link
Contributor

@afbjorklund afbjorklund commented May 31, 2020

I would like to cooperate (minikube with kind), regarding the podman support.

We have added a "podman" driver to minikube next to the docker driver,
and a (varlink) "podman-env" next to the minikube docker-env command.

Since the "kicbase" we are using is based on kindbase, trying to upstream all

@BenTheElder
Copy link
Member Author

@BenTheElder BenTheElder commented May 31, 2020

xref: #1639, thanks for the PR! cc @aojea @amwat ^^

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
10 participants