Compatible with Docker for Mac

[mfburnett]

Several Localkube users have requested support for Docker for Mac, instead of relying directly on VirtualBox. Wanted to open an issue to discuss whether this is something we want to pursue.

[dlorenc]

Thanks for starting this discussion. Do you have any more info on how people want to use localkube in docker for mac? We could publish a docker image that contains just localkube that's ready to run.

[mfburnett]

cc/ @zoidbergwill, @nicdoye

[nicdoye]

Docker users can think of each image running directly on their Mac (as if their desktop were a Linux box). It would be great to think we could view or think of pods as being first class objects in the same way (even if they can't really be). Of course, we then have everything else, too, services, load balancers, etc. Quite how you abstract those objects out onto a single, physical, non-Linux OS could be "difficult".

But is this far too complicated? Should I lower expectations, and accept (after re-reading the minikube proposal) that I'm going to be stuck with VirtualBox, which would be a lot easier to implement.

@mfburnett replied to my comments on Twitter, one of which bemoaned the overhead (actually, the heat generated) from using VirtualBox. The advantages of a lightweight implementation like xhyve for lower power consumption when commuting, not burning my legs, and I assume faster bring-up/tear-down are obvious, but is it just too much work?

[dlorenc]

Thanks for clarifying. Running minikube/localkube inside of hyperkit is definitely something we're
considering.

I haven't played with it enough to really say how much work it will be, but I'm going to start looking into it as soon as we get a stable release of minikube out. In the meantime, any help/prototyping here would be greatly appreciated!

cc @rdayal

[dlorenc]

ref kubernetes/kubernetes#23417

[zoidyzoidzoid]

Last time I tried to use localkube with Docker for Mac, there were issues with how localkube assumes a user is on Linux or in a docker-machine (presumably on OSX/Windows). There were also issues with weave and networking on Docker for Mac.

I think these two tickets are Check for boot2docker and localkube problems with Docker for Mac beta.

Also, Docker for Mac has moved to having containers accessible on localhost:PORT now which helps.

[luxas]

Soon, we'll add an option for minikube to run localkube inside a docker container.
Do you think that will work with docker for mac?

The weave networking problems have been fixed.

(I have no mac, so I can't really test this, but I'm trying to understand how it works)

[zoidyzoidzoid]

So because minikube configures its own VM using libmachine for OSX/Windows, Docker for Mac shouldn't really affect it. Interacting with that DOCKER_HOST doesn't work though, but that might be expected:

± eval $(./out/minikube docker-env)
± docker ps
Error response from daemon: client is newer than server (client API version: 1.23, server API version: 1.21)

Using minikube directly the only issue I have had so far is never getting an external IP but that might be expected behaviour on OSX, or in general?

@luxas: I'll happily help test anything on Mac, but that might work. I had some issues last time but that was a while ago.

[rodcloutier]

@dlorenc The expectation of people using docker for mac (or windows) is that we should be able to run a container without having to setup a VM. That is what docker did when they moved from docker-toolbox (docker-machine using a VM) to the "native" application.

I feel that minikube is somewhat repeating this history. I actually, naively tried to use localkube with the docker for mac assuming that it would work since it is using docker. Investigation led me to this project and was somewhat disappointed with the direction for Windows and Mac.

I expect that lots of user are thinking the same way: we now have native docker app, all we need is a container and we are off and running. It could be somewhat of a letdown to see that we are moving forward with one tool only to be brought backward with another tool...

Are there any major technical constraint to move to a container based approach? So far I have seen the DNS mentioned and shielding from docker upgrade. Using a VM doesn't not necessarily solves the docker issue (assuming that we want to connect to the hosted daemon, which might not be the case) since from the official docker documentation:

The coexistence setup (docker-toolbox and Docker for Mac) works as is as long as your VirtualBox VMs provisioned with docker-machine run the same version of Docker Engine as Docker for Mac.

Kmachine has the same problem since the client and server must be in sync but it only controls the server part, a problem docker doesn't have as it controls both parts.

Maybe minikube should be layered and the container could be provided and supported for multiple version of docker and the VM part use that container. We would then have the choice of using one or both parts.

[zoidyzoidzoid]

@rodcloutier Some of your thoughts are covered in the proposal, but I do agree that'd be nice to have it running in the same VM that Docker for Mac has already set up so I can easily access those built and pulled images.

[luxas]

@rodcloutier @zoidbergwill The docker-only setup is in our roadmap, but it's a bit harder to get working than with virtualbox, so it will probably take a bit longer before everything is implemented.

The first piece is here: #119

[vishh]

Unless docker can provide LTS releases, it will be difficult to implement minikube against docker engine directly.
I can't think of reasons why minikube cannot re-use the virtualization technology that docker engine is currently using.

[dlorenc]

Existing Tools

Here are some notes I have, from digging into the various xhyve/hyperkit/docker-machine-dryver-xhyve drivers.

xhyve

The canonical xhyve repo is https://github.com/mist64/xhyve. It's still seeing some activity, but not much. Specifically, this PR to add virtio-9p support seems stalled.

hyperkit

Hyperkit appears to be a fork of xhyve right now, plus some additional code. It has the virtio-9p support merged in, among other improvements.

docker-machine-driver-xhyve

This repo is a Docker Machine driver implementation that supports xhyve. It currently works with minikube, using this PR.

This uses hooklift/xhyve as a go dependency for manipulating xhyve machines. hooklift/xhyve contains a vendored copy of mist64/xhyve, with some patches applied. virtio_9p support seems to be missing, though.

TheNewNormal/libxhyve

This is a fork of hooklift/xhyve, with patches applied to rebase on top of hyperkit. It has virtio_9p support merged in.

Current Status

I'm currently experimenting with getting docker-machine-driver-xhyve updated to use TheNewNormal/libxhyve instead of hooklift/xhyve.

Long term, we should include all the relevant code in minikube via vendoring, and not rely on users adding extra drivers to their path.

[jimmidyson]

Looking at embedding drivers tonight, doesn't look too big a problem & will end up with one sweet binary for distribution.

[dlorenc]

Here's the docker-machine-driver prototype on TheNewNormal/libxhyve: https://github.com/dlorenc/docker-machine-driver-xhyve/tree/hyperkit

It appears to work fine.

[dlorenc]

Note: The link I posted earlier only mostly worked. I had to hack the vendor directory up a bit:

1. To fix a Godeps bug where cgo code isn't handled properly.
2. To get virtio-9p support working again. The version of pci_virtio_9p.c in hyperkit doesn't appear complete, it relies on another 9p server running.

[jimmidyson]

Could you link to the godep issue? I don't have a problem with cgo, godep & KVM plugin so just wondering what it is.

Is 9p support a deal breaker? Can it be added later?

[dlorenc]

Here's the issue: tools/godep#422

It looks like godep works unless the cgo files are in another directory. We don't need 9p support to start with, but it's been one of the biggest areas the xhyve forks differ. As long as we pick the right fork we'll be fine.

[dlorenc]

With #248, we'll have experimental xhyve support in the next release of minikube. There's still a bit more to do to get the full "docker for mac" experience, though.

Let's keep using this as an umbrella issue for vpnkit integration and possibly osxfs integration.

[konobi]

You could run docker-in-docker for both Docker for Mac and Docker for Windows (linux too of course). This would give localkube much more fine grain control over how it's own containers work. No doubt you'd need to start the initial container with a bunch of CAP settings and ensuring that the docker daemon socket is mounted inside the container so that localkube can switch between it's own daemon instance and the hosts daemon instance.

[jimmidyson]

I guess the question comes how much we want a consistent experience across OSes vs an optimised one per OS.

[solsson]

vyshane/kid runs Kubernetes 1.3.0 in Docker for Mac

[numbsafari]

Let me know if you'd like to use anyone as a guinea pig or if there's anything I can do to help. VirtualBox has generally been a royal pain and I'd love to help getting things working under xhyve.

[numbsafari]

s/xhyve/hyperkit/ ... I'm pretty sure the docker folks have gotten everyone working on xhyve to work on hyperkit.

[VojtechVitek]

@dlorenc @jimmidyson I really appreciate your work on this guys. Do you have any updates RE: latest Hyperkit / xhyve drivers?

Is there any way we can make minikube work on MacOS without having Virtualbox installed?

$ minikube start --vm-driver=xhyve --container-runtime=docker --show-libmachine-logs --v=10 --alsologtostderr
I1019 19:24:15.138185   61371 notify.go:111] Checking for updates...
Starting local Kubernetes cluster...
Found binary path at /usr/local/bin/docker-machine-driver-xhyve
Launching plugin server for driver xhyve
Plugin server listening at address 127.0.0.1:51272
() DBG | operation not supported by device
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetMachineName
(minikube) Calling .DriverName
Running pre-create checks...
(minikube) Calling .PreCreateCheck
(minikube) DBG | ===== Docker Machine xhyve Driver Version 0.2.2 (Homebrew) =====
(minikube) DBG |
(minikube) DBG | executing: /usr/local/bin/VBoxManage -v
(minikube) DBG | STDOUT:
(minikube) DBG | STDERR: /usr/local/bin/VBoxManage: line 2: /Applications/VirtualBox.app/Contents/MacOS/VBoxManage: No such file or directory
(minikube) DBG | /usr/local/bin/VBoxManage: line 2: exec: /Applications/VirtualBox.app/Contents/MacOS/VBoxManage: cannot execute: No such file or directory

[dlorenc]

Hey,

This looks like machine-drivers/docker-machine-driver-xhyve#134

You don't need virtualbox installed to use the xhyve driver. This is an unfortunate bug though, because older versions of virtualbox were incompatible with xhyve. It was a bug in virtualbox, vut if you attempted to use xhyve at the same time it could cause a kernel panic.

The xhyve driver is trying to be defensive here, and sees that you have vboxmanage on your path, so it assumes you have virtualbox installed. It then tries to check the version of virtualbox to make sure it won't cause a kernel panic.

This is usually caused by partially uninstalling virtualbox, so vboxmanage is still on your path but the rest of virtualbox is gone.

Could you try the workaround in the bug I linked?

[VojtechVitek]

@dlorenc thank you, that worked! The VBox* binaries in my $PATH were left-overs after (not really successful) uninstall of VirtualBox ~1y ago.

$ minikube status
minikubeVM: Running
localkube: Running

[r2d4]

Closing this. We currently have xhyve support and we're tracking bug fixes there

[zoidyzoidzoid]

Since the export DOCKER_API_VERSION stuff, the xhyve support works great, and there are no frustrating errors about the client version being too new.

[dobegor]

@r2d4 why is this issue considered closed?
I.e. I've got a latest version of Docker for Mac, which embeds hyperkit, and still latest minikube complains that xhyve is not in $PATH.

Well, it may not be able to find the binary itself since Docker for Mac now ships in .app bundle.
But it still wants to create a separate VM instead of using one created by Docker for Mac, which was the whole point of this issue.

Is there any way to accomplish this?

[VojtechVitek]

@dobegor take a look at #99 (comment)

[dobegor]

@VojtechVitek sorry, how is this related? I don't have VBox installed.

[dobegor]

I've just downloaded Docker for Mac and Minikube on fresh OS X. Results are the same: minikube doesn't work even with --driver xhyve option.

[dlorenc]

Can you attach the output of minikube start --vm-driver=xhyve?

Minikube doesn't rely on Docker for Mac at all, and instead uses it's own copy of xhyve which must be installed via brew install docker-machine-driver-xhyve

[dobegor]

@dlorenc, sure

$ minikube start --vm-driver xhyve
Starting local Kubernetes cluster...
E0203 18:25:45.240196    1866 start.go:107] Error starting host: Error creating new host: Driver "xhyve" not found. Do you have the plugin binary accessible in your PATH?.

 Retrying.
E0203 18:25:45.240750    1866 start.go:113] Error starting host:  Error creating new host: Driver "xhyve" not found. Do you have the plugin binary accessible in your PATH?

$ docker --version
Docker version 1.13.0, build 49bf474

Downloaded from: here

[dlorenc]

Thanks! If you follow the installation step here you should be all set:
https://github.com/kubernetes/minikube/blob/master/DRIVERS.md#xhyve-driver

[deinspanjer]

I have the xhyve-driver installed and set up fine and I've been able to run minikube successfully, but I'll echo the question about why this issue has been closed. Wasn't the whole point of the issue to be compatible with Docker for Mac rather than just using the same method for spinning up a VM?

Is there an important reason why we can't have minikube use the existing Docker for Mac VM and docker-engine in it rather than having to create a completely new VM?

Practically, it seems this would be more convenient to allow reuse of images build in the Docker for Mac VM and fewer total resources consumed.

[dobegor]

@deinspanjer that's a question for me, too. I've managed to get it up and running thanks to @dlorenc's advice, but this spins an additional VM and I can't use docker ... commands with it since it's pointing to another VM created by Docker for Mac.

@r2d4 may we ask you to reopen this issue?

[deinspanjer]

I just read through the proposal doc ( https://github.com/kubernetes/community/blob/master/contributors/design-proposals/local-cluster-ux.md ), and I suspect the issue is what is called out in the "Bring your own docker" section. If we rely on the existing VM and the docker-engine that is installed in it, then we expect the minikube code to always be compatible with that version of the VM and that version of the docker-engine. If the user upgrades their Docker for Mac install and the upgrade has some incompatibility, it will also break their entire minikube environment.

It is possible to get docker ... commands working with the minikube environment by running $(minikube docker-env), but this doesn't resolve the one big outstanding issue that I see, and that is leaving behind all the container images you have built and stored in your Docker for Mac VM.

Is there any good way to be able to share and reuse those in the minikube environment?

[atombender]

I can understand why the Kubernetes team would want an "all batteries included (for ever)" solution, but there are many arguments for not wanting this.

The easiest solution would be for Minikube to be able to run in a mode where it simply reuses Docker for Mac, at the expense of requiring that the user manage any compatibility issues.

[iameli]

Alright, I have a weird prototype a Docker-for-Mac based Kubernetes cluster running over here. So far it's performing quite well.

Very hacky at the moment, but I'm curious if this is something that Minikube would like interested in integrating eventually.

[dlorenc]

Hey,

Thanks for all the feedback! Here are some responses inline.

Is there an important reason why we can't have minikube use the existing Docker for Mac VM and docker-engine in it rather than having to create a completely new VM?

Practically, it seems this would be more convenient to allow reuse of images build in the Docker for Mac VM and fewer total resources consumed.

There are a few issues with re-using the same VM. The biggest is that Docker for Mac doesn't expose or document any interfaces for reuse. We'd essentially have to reverse engineer it, which could work, but would be brittle in the long run.

Alternatively we could run most of the Kubernetes components inside Docker itself (which @iameli is doing above), but many of the k8s components still have trouble when running inside a container.

The easiest solution would be for Minikube to be able to run in a mode where it simply reuses Docker for Mac, at the expense of requiring that the user manage any compatibility issues.

We're considering supporting a mode like this (@aaron-prindle is prototyping it now) in #1173.

[iameli]

...but many of the k8s components still have trouble when running inside a container.

Y'know, I get this response a lot whenever I get anywhere near kubelet's --containerized flag, and I'm sure it's based in Kube knowledge that I don't possess. But anecdotally, I've had a containerized cluster running on an Ubuntu host for the last six months, using it for full-time Streamplace development, with almost no problems to report. Do you know of anywhere where the limitations are more specifically enumerated?

The only thing I've been able to find specifically is this list of preconditions for NsenterMounter to work properly, which doesn't seem too onerous.

[dlorenc]

Unfortunately I don't have a great link. A long time ago I tried running the kubernetes conformance tests against a containerized cluster, and there were quite a few failures. If you're interested in finding out more, that would be a good place to start.