[WIP] New Driver: xhyve

[zchee]

This pull request is WIP. and "Pull Request Driven Development".
I'm writing now.

mist64/xhyve

The xhyve hypervisor is a port of bhyve to OS X. It is built on top of Hypervisor.framework in OS X 10.10 Yosemite and higher, runs entirely in userspace, and has no other dependencies.
It can run vanilla Linux distributions and may gain support for other guest operating systems in the future.

but, @SvenDowideit said boot2docker/boot2docker#922 (comment).
If docker core team had been working, I will withdraw the PR.

Milestones

• Tentative use custom boot2docker
• Get xhyve VM IP from UUID
• Some share folder technology or NFS synced folder
• Support specified CPUs count & Memory size
• Support the all commands that docker-machine has implemented
• Daemonize xhyve
• Full usage documents

[zchee]

/cc @ehazlett @nathanleclaire

[zchee]

@tiborvass said,
boot2docker/boot2docker#922 (comment).
yes, xhyve is super experimental project. I know.

[far-blue]

This would be great to see supported in docker-machine. Being a single lightweight binary, not having to install kernel extensions, using hypervisor tech officially supported by Apple and not having Oracle looming over it is a powerful combination :)

Also note that RancherOS is now supported in 0.3.0-rc2 in VirtualBox as an alternative to Boot2Docker so you might want to talk to them about support for xhyve as well.

[emmanuel]

@ailispaw has already wrapped the boot2docker ISO & initrd with a simple Makefile & startup script for xhyve: https://github.com/ailispaw/boot2docker-xhyve

Not sure if that info is at all useful w/r/t docker-machine, but it lets me use xhyve today for Docker on OS X, which means no more VirtualBox for me 😄.

[tiborvass]

@emmanuel that seems perfect to be reused as a machine driver :)

[ehazlett]

👍

[nathanleclaire]

👍

[zchee]

@ehazlett @nathanleclaire :)

It can be implemented in a little later.
Once dockercon is closing, the review to please :)

[zchee]

@emmanuel Yes, This PR inspired by boot2docker-xhyve :)

[holidayworking]

👍

[zchee]

Current status.

• Tentative use custom boot2docker.iso
• Get xhyve VM IP from UUID
• NFS synced folder
• Support specified CPUs count & Memory size

Publish code is please wait a little longer.

[tiborvass]

@zchee I don't know how you pushed, but this PR has just one empty commit: zchee@3e1268c

Let us know if you need help.

[tiborvass]

Also I don't think we want NFS for synced folder. It's a more general problem and this PR shouldn't try to fix that now.

[zchee]

@tiborvass OK. I'll push later.
I'm out now, after half a day

Also I don't think we want NFS for synced folder. It's a more general problem and this PR shouldn't try to fix that now.

Yes, I now. Tentative.

[nickschuch]

Do you happen to have a link to the NFS synced folder problem? Curious about how this is going to be handled.

[zchee]

Not works now.

Tentative dependency tools

• https://github.com/zchee/xhyve
  • Supported virtio-tap custom build xhyve
• https://github.com/zchee/boot2docker-xhyve/releases/download/uuid2mac%2Fv1.0.0/uuid2mac
  • Get MAC address from xhyve flaged UUID (deprecated)
• https://github.com/zchee/boot2docker
  • Custom boot2docker iso (not publish now)
    • Enable NFS share (Tentative)
    • PXE command hack

Execute OS buildin command

• sudo (https://github.com/mist64/xhyve#todo)
• dd
• hdiutil
• uuidgen (deprecated. will rewrite go)

Current this code has a lot of improvements.
And, I'm for the first time to write the golang code.
I'm improving now.

@ALL Actively please advice me!

[zchee]

@nickschuch I'm still thinking... :(

[nikolay]

👍 This will be huge!

[cbarratt]

This would be amazing to use, especially if Kitematic supports it!

[timfallmk]

@zchee Made some very minor comments on you code, but otherwise looks great! What's left to do?

[timfallmk]

@zchee @tiborvass I've added a check for Virtual Box 5 to PreCreate for xhyve. I checked it against a dummy slug, but I don't have virtual box installed, so someone check me. timfallmk@3feb245

Note: Since the vbm functions are nonexported, I had to copy over the functions I needed to utils. Not an ideal way to do things. An export wrapper for vbm would be really nice.

[estevaoam]

This is amazing 👍

[marcelaraujo]

+1

👏 👏 👏

[zchee]

@timfallmk Sorry, missed merge.
I will fix day after tomorrow.

[tiborvass]

@zchee I would rather compile xhyve into machine. I started working on some go bindings to xhyve, hopefully I'll have something next weekend. That way we won't need to shell out to xhyve and won't require the users to install (aka compile) xhyve at all.

[timfallmk]

@tiborvass Could you share you progress when you get the chance so we can help out?

[tiborvass]

@timfallmk Here I uploaded it https://github.com/tiborvass/xhyve-bindings it's still very early but the main example should work.

[timfallmk]

@tiborvass Looks interesting, let me know if I can help. Would an appropriate stopgap measure be using a staticly compiled binary to distribute for xhyve installs?

[tiborvass]

@timfallmk that's the first thing I tried, but I couldn't figure out how to do LTO with static binaries and the Go linker. But def, that would be the best thing. Also you would have to rewrite (or just copy to cgo) all the logic in xhyve.c outside of main(). Feel free to contribute there, all suggestions welcome for whatever makes it possible to have no dependencies in Go :)

[ehazlett]

Hi , thanks for your efforts and persistence in submitting this driver. We are extremely excited that there is so much interest in Docker Machine and we really appreciate your interest. However, at this time it is proving to be extremely difficult for us to keep up with reviewing and testing each of these drivers for inclusion in the Machine core. We really want to switch to a more pluggable model, as well as polish up a few things about the driver model which need to be changed to ensure a smooth and sustainable future.

Therefore, we will be moving to a plugin model for 0.5 and would love to have you involved in the design and development process. We are closing the outstanding driver PRs at this time, but please keep the code. We will stick closely to the current driver interface and you should be able to re-use a lot (if not all) of the existing driver along with the new plugin model. We will be moving all of the drivers which are merged directly into Machine today to the plugin model when it is available, so there will be no special treatment of those, and there will be documentation outlining the process of developing and using a Docker Machine driver plugin.

With all of that being said, we want to apologize for the lack of feedback on your pull request. As contributors ourselves, we understand that being left in limbo is no fun. We would have liked to address this sooner, and in the future we will be more responsive around these kinds of issues.

Once again, we thank you for the contribution and the tremendous support. Keep hacking strong!

If you want to contribute to the design of the plugin model, we'd love to get your input on this issue where we will be planning it:

#1626

@ehazlett & @nathanleclaire

[timfallmk]

Ouch. This was ready to be merged. 😦

[tiborvass]

@timfallmk just look at that issue and the number of "ready-to-be-merged" drivers ;)

[timfallmk]

Sounds like you just need more warm bodies! (cough cough)

/kidding

[robmorgan]

damn, this was cool. I'm looking forward to seeing it existing at some form in the future!

[nathanleclaire]

@zchee @timfallmk and/or anyone interested in working on this, I have a crude version of the plugins model working now, and would like some help porting this to the new model. I have the plugin successfully compiled and running from the machine binary but it always fails to get the DHCP lease (in fact, /var/db/dhcpd_leases doesn't exist at all on my computer).

Hit me up (nathan@docker.com) if you want to work on it, we want to start getting feedback on plugins ASAP. I intend to have a PR for plugins in soon as well, but it depends on the large libmachine work, so it won't be very readable for a while.

[timfallmk]

Hey Nathan,
I'm still in. I've been working on related projects recently.

On Thu, Sep 10, 2015 at 11:46 AM Nathan LeClaire notifications@github.com
wrote:

@zchee https://github.com/zchee @timfallmk
https://github.com/timfallmk and/or anyone interested in working on
this, I have a crude version of the plugins model working now, and would
like some help porting this to the new model. I have the plugin
successfully compiled and running from the machine binary but it always
fails to get the DHCP lease (in fact, /var/db/dhcpd_leases doesn't exist
at all on my computer).

Hit me up (nathan@docker.com) if you want to work on it, we want to start
getting feedback on plugins ASAP.

—
Reply to this email directly or view it on GitHub
#1358 (comment).

[timfallmk]

Nice. Let me slap together a branch for you and we can start sending some
work back and forth!

On Thu, Sep 10, 2015 at 3:22 PM, Tim Fall timfall@gmail.com wrote:

Hey Nathan,
I'm still in. I've been working on related projects recently.

On Thu, Sep 10, 2015 at 11:46 AM Nathan LeClaire notifications@github.com
wrote:

@zchee https://github.com/zchee @timfallmk
https://github.com/timfallmk and/or anyone interested in working on
this, I have a crude version of the plugins model working now, and would
like some help porting this to the new model. I have the plugin
successfully compiled and running from the machine binary but it always
fails to get the DHCP lease (in fact, /var/db/dhcpd_leases doesn't exist
at all on my computer).

Hit me up (nathan@docker.com) if you want to work on it, we want to
start getting feedback on plugins ASAP.

—
Reply to this email directly or view it on GitHub
#1358 (comment).

[ehazlett]

I would like to get the libmachine and plugins PRs in the machine repo ASAP so we can have all of the current driver maintainers and proposed to have the ability to do this as well. I think the longer we do this in separate repos it will get really messy.

[thaJeztah]

@nathanleclaire this is the POC you're working on? https://github.com/nathanleclaire/docker-machine-xhyve

[zchee]

Hi, all.
long time no see.
Uh... nathan met in Japan :)

I think...

First, the #1626 discussion is settled is likely to take a long time.

Second, xhyve last commit on Jul 27, 2015.
It's quite a long time ago.
but, @bonifaido wrote Migrate vmm_callout to libdispatch (Grand Central Dispatch?) on xhyve/src/vmm/vmm_callout.c.
It seems the development is progressing behind the scenes.

And, NextBSD project also has started.

This is the darwin-support branch. It adds support for a subset of the mach APIs
to enable FreeBSD to run some of the OSX daemons.

And more than anything, there is no possibility that apple is not anything action for Hypervisor framework.

While the timing is also feel like not now...
but, for now, I will rewrite the reference to @tiborvass's xhyve-bindings and @nathanleclaire's docker-machine-xhyve.

/var/db/dhcpd_leases doesn't exist at all on my computer.

In my case, it is created at the moment you start the xhyve, it was possible to get the IP.

Sorry bad English.
Thanks.

[zchee]

For all.

Now, Moved this PR.
See nathanleclaire/docker-machine-xhyve#2 also https://github.com/zchee/docker-machine-xhyve