Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support the rkt/minikube-iso #140

Merged
merged 8 commits into from
Oct 31, 2016
Merged

Support the rkt/minikube-iso #140

merged 8 commits into from
Oct 31, 2016

Conversation

zchee
Copy link
Member

@zchee zchee commented Oct 8, 2016

Support the rkt/minikube-iso.

  • Remove go generate mv .o .syso, use #cgo LDFLAGS: -lfoo.o
  • Temporary fix ramdisk_start to twice size.
    • It would be incorrect but can boot large size kernel.
    • Tested tinycore, debian testing, ubuntu yakkety, minikube-iso and boot2docker
  • Support any filename of kernel executable and initrd image.
    • Because of different filename by distro, tools or etc. such as build by Buildroot.

@zchee
Copy link
Member Author

zchee commented Oct 8, 2016

/cc @dlorenc @r2d4
Please test it use minikube with
https://github.com/coreos/minikube-iso/releases/download/xhyve/minikube-xhyve.iso
minikube-v0.0.5.iso is bzImage and initrd so can not yet boot.
If try build, GO_BUILD_TAGS='lib9p qcow2' make V=1. not "GO_BUILD_TAG", add S. sorry.

/cc @s-urbaniak
Added d.Vmlinuz and d.Initrd.
They are initialized d.SetConfigFromFlags if use docker-machine. Assumed boot2docker.iso if empty.
According to some document, /boot directory is a specification. So hardcode at
https://github.com/zchee/docker-machine-driver-xhyve/blob/minikube-iso_kexec/xhyve/xhyve.go#L606-L607

Is there no problem this fix?

@zchee
Copy link
Member Author

zchee commented Oct 8, 2016

I will release 0.3.0 version after the merge it and some fix.

@zchee zchee force-pushed the minikube-iso_kexec branch 2 times, most recently from d0a465c to 7d55ee7 Compare October 9, 2016 00:11
@s-urbaniak
Copy link

s-urbaniak commented Oct 11, 2016

@zchee LGTM! I'll prepare a PR against minikube to configure the driver correctly with the minikube-iso image.

@zchee
Copy link
Member Author

zchee commented Oct 11, 2016

@s-urbaniak Thanks for test :)
Did you say PR means for xhyve-driver? minikube?
If minikube, I was confirmed successful with this branch, just for reference.
zchee/minikube@b646163

@s-urbaniak
Copy link

@zchee I would prepare a PR against minikube like you did in zchee/minikube@b646163 :-) Although looking at your change I am wondering how to "autodetect" whether it is a minikube-iso or boot2docker, maybe by looking at the --iso-url string.

@zchee
Copy link
Member Author

zchee commented Oct 11, 2016

@s-urbaniak Ah, yeah.
It supports minikube-iso only(Sorry, it's my quick hack for minikube-iso test lol), So if set --iso-url with boot2docker.iso from minikube side, will fail.

It would be better to implement the processing. Which is implement parse of iso? xhyve side? minikube?

Edit: Do you think which one is better?

@s-urbaniak
Copy link

Hmm ... I just tried out this PR, and I get this error:

$ minikube --vm-driver=xhyve start --show-libmachine-logs --v=5 --alsologtostderr --stderrthreshold 5
W1012 14:21:13.055893    4662 root.go:127] Error reading config file at /Users/sur/.minikube/config/config.json: open /Users/sur/.minikube/config/config.json: no such file or directory
I1012 14:21:13.056541    4662 notify.go:111] Checking for updates...
Starting local Kubernetes cluster...
Found binary path at /Users/sur/src/go/src/github.com/zchee/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
Launching plugin server for driver xhyve
Plugin server listening at address 127.0.0.1:49533
() DBG | operation not supported by device
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetMachineName
(minikube) Calling .DriverName
Creating CA: /Users/sur/.minikube/certs/ca.pem
Creating client certificate: /Users/sur/.minikube/certs/cert.pem
Running pre-create checks...
(minikube) Calling .PreCreateCheck
(minikube) DBG | ===== Docker Machine xhyve Driver Version 0.2.3 (7d55ee7) =====
(minikube) DBG | 
(minikube) DBG | executing: /usr/local/bin/VBoxManage -v
(minikube) DBG | STDOUT: 5.1.6r110634
(minikube) DBG | 
(minikube) DBG | STDERR: 
(minikube) Calling .GetConfigRaw
Creating machine...
(minikube) Calling .Create
(minikube) Downloading /Users/sur/.minikube/cache/boot2docker.iso from file:///Users/sur/.minikube/cache/iso/minikube-0.7.iso...
(minikube) Creating VM...
(minikube) Extracting kernel and initrd from boot2docker.iso...
(minikube) DBG | Mounting boot2docker.iso
(minikube) DBG | executing: &{/usr/bin/hdiutil [hdiutil attach /Users/sur/.minikube/machines/minikube/boot2docker.iso -mountpoint /Users/sur/.minikube/machines/minikube/b2d-image] []  <nil> 0xc420086008 0xc420086010 [] <nil> <nil> <nil> <nil> <nil> false [] [] [] [] <nil> <nil>} attach /Users/sur/.minikube/machines/minikube/boot2docker.iso -mountpoint /Users/sur/.minikube/machines/minikube/b2d-image
(minikube) /dev/disk5                                           /Users/sur/.minikube/machines/minikube/b2d-image
(minikube) DBG | Extracting kernel into /Users/sur/.minikube/machines/minikube
E1012 14:21:23.427927    4662 start.go:87] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: open /Users/sur/.minikube/machines/minikube: is a directory. Retrying.

PS: The master branch works.

- qcow2: remove go generate 'mv .o .syso', use '#cgo LDFLAGS: -lfoo.o'

- kexec: temporary fix ramdisk_start to twice size
  - it would be incorrect, but can boot large size kernel.
  - Tested tinycore, debian testing, ubuntu yakkety, minikube-iso and
    boot2docker

Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com>
- Support modifiable kernel executable and initrd image filename,
  because of different filename by distro, tools. such as buildroot

Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com>
Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com>
@zchee
Copy link
Member Author

zchee commented Oct 12, 2016

@s-urbaniak Ah, really...? work fine this pull request :(
but I was fixed several times this pull request, and added some better log messages.
Could you build again?


Edit: pre-compile binary and test iso

http://get.zchee.io/bin/docker-machine-driver-xhyve

shasum -a 256 ./bin/docker-machine-driver-xhyve
44cb25be703e416202ed24189c4b26fbc37342de489bbf53fc99140ae9b14acc  ./bin/docker-machine-driver-xhyve

http://get.zchee.io/bin/minikube

shasum -a 256 minikube
869880dbc61074fc49d018549274e877cb117698598f673338e5d0e9a32d5114  minikube

http://s3.zchee.io/iso/minikube-virtblk.iso

shasum -a 256 minikube-virtblk.iso
93301b7231f94bc4b1662ac486dd09e01c3b34b45947346f759370300861aeef  minikube-virtblk.iso

@zchee
Copy link
Member Author

zchee commented Oct 18, 2016

/cc @dlorenc @r2d4 @s-urbaniak
Added latest docker/hyperkit including fixed kexec.c.
PTAL.

- kexec.c commit message
  - Use the initrd_addr_max value from the kernel header if the
    kernel version is 2.03 or newer. Use hardcoded(0x37ffffff) value
    if older.
  - Compare the initrd_max and memory.size, and re-assign if
    initrd_max is larger.
  - Add ALIGNDOWN for ramdisk_start, uses prevent to go past the
    address limit.
  - Based on qemu/hw/i386/pc.c

Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com>
@r2d4
Copy link
Contributor

r2d4 commented Oct 19, 2016

@zchee I had some trouble creating the vm. I'm get the same error as s-urbaniak above

E1018 19:57:31.013713 1536 start.go:93] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: open /Users/mrick/.minikube/machines/minikube: is a directory

@zchee
Copy link
Member Author

zchee commented Oct 20, 2016

@r2d4 Hmm... is this binary build from source with this PR?
I'll try test later.

@zchee
Copy link
Member Author

zchee commented Oct 20, 2016

@s-urbaniak @r2d4 Ah, repro on latest minikube
I'll fix it. sorry.

Edit:
In @r2d4 case, maybe needs zchee/minikube@b646163
In @s-urbaniak case, maybe that is old binary I think.

@zchee
Copy link
Member Author

zchee commented Oct 25, 2016

@dlorenc @r2d4 @s-urbaniak
I can not release the new version because waiting for merge this pull request.
How is the progress on minikube and minikube-iso?

@s-urbaniak
Copy link

@zchee since I wasn't able to get the xhyve driver running locally with this PR, I was not able to progress on the minikube side. When I'll be back in the office next week, I can retry using this PR, and the latest minikube on master.

@dlorenc
Copy link
Collaborator

dlorenc commented Oct 28, 2016

Hey,

I'm trying to build this locally and I'm getting this error:

duplicate symbol _caml_tuplify5 in:
    mirage_block_ocaml.syso
    vendor/github.com/zchee/libhyperkit/mirage_block_ocaml.o
duplicate symbol _caml_tuplify7 in:
    mirage_block_ocaml.syso
    vendor/github.com/zchee/libhyperkit/mirage_block_ocaml.o
duplicate symbol _caml_tuplify8 in:
    mirage_block_ocaml.syso
    vendor/github.com/zchee/libhyperkit/mirage_block_ocaml.o
ld: 14678 duplicate symbols for architecture x86_64

Any ideas?

@zchee
Copy link
Member Author

zchee commented Oct 28, 2016

@dlorenc It seems to exists mirage_block_ocaml.o and mirage_block_ocaml.syso file on any place. maybe on the vendor/github.com/zchee/libhyperkit.
Please remove mirage_block_ocaml.syso, or just make clean before building.
https://github.com/zchee/docker-machine-driver-xhyve/blob/6e75aa47e6e89aae20a83de923019dc774f6f5a2/Makefile#L219-L220

@dlorenc
Copy link
Collaborator

dlorenc commented Oct 28, 2016

Ah nice, make clean wasn't working but I recloned the repository and got it to build.

$ minikube version
minikube version: v0.12.1
$ GO_BUILD_TAGS='lib9p qcow2' make V=1 install

but I'm now getting the same error as @r2d4.

@dlorenc
Copy link
Collaborator

dlorenc commented Oct 28, 2016

I think I found the issue. I'll send a few PRs in a bit.

@zchee
Copy link
Member Author

zchee commented Oct 28, 2016

@dlorenc Ah, I was talking on the slack with @r2d4, This pull request needs this change on minikube side.
zchee/minikube@b646163
Because I added initrd and vmlinuz struct variable.

If call xhyve-driver by minikube, will initrd and vmlinuz is empty. So occur error.

However, this problem might be solved with adding initialize method.
It means like

Sorry, I still don't understand minikube internal.
Where do I should add the initialize method for having a backward compatibility of minikube?

@dlorenc
Copy link
Collaborator

dlorenc commented Oct 28, 2016

Where do I should add the initialize method for having a backward compatibility of minikube?

Maybe in here: https://github.com/kubernetes/minikube/blob/master/pkg/minikube/cluster/cluster_darwin.go#L63

?

@zchee
Copy link
Member Author

zchee commented Oct 28, 2016

@dlorenc Ah, sorry. Maybe I misunderstood and mistaken...
Please give me some time, I'll fix this pull request.

@dlorenc
Copy link
Collaborator

dlorenc commented Oct 28, 2016

Sure! Let me know if I can help.

@zchee
Copy link
Member Author

zchee commented Oct 28, 2016

@dlorenc Thanks.

BTW, I want to local test use original minikube-alpha(?) iso file.
That's has been released? or When scheduled for release?

@dlorenc
Copy link
Collaborator

dlorenc commented Oct 28, 2016

Not sure what you mean. You can grab the rkt/systemd ISO at the URL in the doc here: https://github.com/kubernetes/minikube/blob/master/README.md#using-rkt-container-engine

@zchee
Copy link
Member Author

zchee commented Oct 28, 2016

@dlorenc It means http://storage.googleapis.com/minikube/iso/buildroot/minikube-v0.0.5.iso ?
I tested, but that has not CONFIG_VIRTIO_BLK=y.
coreos/minikube-iso#28

In other words, When you plan to minikube support of qcow2 disk image?

@s-urbaniak
Copy link

Indeed the virtio config is not present in the 0.0.5 release yet, as coreos/minikube-iso#28 was merged after the release. dlorenc let me know if I can help out with releasing a 0.0.6, which would include the above kernel config.

@dlorenc
Copy link
Collaborator

dlorenc commented Oct 29, 2016

@r2d4 just released 0.6 yesterday.

@zchee
Copy link
Member Author

zchee commented Oct 31, 2016

@dlorenc thanks for release 0.6.

@zchee
Copy link
Member Author

zchee commented Oct 31, 2016

@dlorenc Merged your two PR. thanks.
Ready to merge it?

@dlorenc
Copy link
Collaborator

dlorenc commented Oct 31, 2016

Yeah, doing some final sanity testing now :)

@r2d4 could you try this out today too?

@zchee
Copy link
Member Author

zchee commented Oct 31, 2016

@dlorenc Got it. I'll re-test with docker-machine and minikube.

@dlorenc
Copy link
Collaborator

dlorenc commented Oct 31, 2016

Nice! Everything works for me. Merge whenever :)

@zchee
Copy link
Member Author

zchee commented Oct 31, 2016

@dlorenc

@r2d4 could you try this out today too?

Do I need to wait?

@dlorenc
Copy link
Collaborator

dlorenc commented Oct 31, 2016

Nah, it should be safe to merge now.

@zchee zchee merged commit 4012b0c into master Oct 31, 2016
@zchee zchee removed the in progress label Oct 31, 2016
@zchee
Copy link
Member Author

zchee commented Oct 31, 2016

Merged. Thanks!

@r2d4
Copy link
Contributor

r2d4 commented Oct 31, 2016

It works on my machine also 👍

@dlorenc dlorenc deleted the minikube-iso_kexec branch October 31, 2016 21:32
@zchee
Copy link
Member Author

zchee commented Oct 31, 2016

@dlorenc @r2d4 @s-urbaniak
I'll release next version after some fix (e.g. add ocaml setup to travis.yml)
Thanks for a lot of review and test.

@s-urbaniak
Copy link

@zchee 👍 thanks a lot!!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants