Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/builders: tracking bug for macOS virtualization #9495

Open
bradfitz opened this issue Jan 3, 2015 · 32 comments

Comments

@bradfitz
Copy link
Member

commented Jan 3, 2015

I'd like to get OS X builders running in VMs too.

(Related to #9492 and https://golang.org/s/builderplan)

Looks like virtualizing OS X 10.7+ is legal (but not 10.6) as long as it's only 1 copy, and on official Apple hardware. Considering that we already run Go builders on official Mac hardware in the office, we can continue to do so, but with a VM solution.

And looks like VMWare Fusion has the "vmrun" command, documented at http://www.vmware.com/pdf/vix162_vmrun_command.pdf , so we can write a little API server that runs on the OS X host and calls vmrun.

/cc @adg

@bradfitz bradfitz added the builder label Jan 3, 2015

@bradfitz bradfitz changed the title builders: track bug for OS X virtualization builders: tracking bug for OS X virtualization Jan 3, 2015

@c4milo

This comment has been minimized.

Copy link
Member

commented Jan 3, 2015

You can also use the VIX bindings I wrote just in case vmrun falls short for any reason: https://github.com/hooklift/govix

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Jan 5, 2015

@c4milo, any chance you want to own this whole bug? :)

@c4milo

This comment has been minimized.

Copy link
Member

commented Jan 5, 2015

hahaha, @bradfitz let me look into it in detail before making any commitments.

@c4milo

This comment has been minimized.

Copy link
Member

commented Jan 5, 2015

@bradfitz alright, how were you envisioning the little API server you mentioned above?

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Jan 5, 2015

I'd need API calls to:

  • start a pre-configured OS X VM, and return me an ID for it (so I can delete it later), and IP:port I can use to hit its port 80.
  • an API call to destroy that VM

That's it. The VM would need to run a couple line shell script on boot.

@c4milo

This comment has been minimized.

Copy link
Member

commented Jan 5, 2015

Working on it.

@c4milo

This comment has been minimized.

Copy link
Member

commented Jan 5, 2015

That's it. The VM would need to run a couple line shell script on boot.

As root?

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Jan 5, 2015

@c4milo, yeah, root works. I can drop down to a less-privileged user if I need to. But some of our net tests require root, or at least more capabilities.

The script you'll want to run as root is just:

#!/bin/bash
set -e
curl --fail -o /usr/local/bin/buildlet https://storage.googleapis.com/go-builder-data/buildlet.darwin-amd64
chmod +x /usr/local/bin/buildlet
exec /usr/local/bin/buildlet

That should work already. Be careful with that server: it is a remote code execution daemon. Don't run it on your real machine if it's exposed to the internet.

But you should be able to hit it in a VM and see it reply "I am a buildlet" or whatever at its root path.

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Jan 5, 2015

(and don't bake in the buildlet to the VM image... only that script. That lets us update the buildlet over time easier than updating the whole VM image)

@c4milo

This comment has been minimized.

Copy link
Member

commented Jan 5, 2015

Got it.

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Jan 5, 2015

Also, tell me what to buy. :)

VMWare Fusion 7 Pro? http://store.vmware.com/store/vmware/en_US/DisplayProductDetailsPage/ThemeID.2485600/productID.304322800

I have a iMac 5K running Yosemite and a Macbook Pro running Mavericks.

Ideally we'll want to test the Go builders with OS X 10.8, 10.9, and 10.10. Maybe 10.7 too. So can I just buy those OS X discs from Apple somehow if I can no longer find my original CDs? Or do I need "Server CDs"? I know you're not a lawyer, and I'm not asking for legal advice. We're definitely not looking to avoid paying for things. I'm happy giving Apple plenty of (more) money here... I just want to know technically what's required, once we pay for all the parts the Apple-sanctioned ways.

@c4milo

This comment has been minimized.

Copy link
Member

commented Jan 5, 2015

I'm using VMWare Fusion 7 Pro. I was also thinking in using OSX Server as the guest OS, since it seems to boot faster, but I might need to double check that once I start testing the API.

@adg

This comment has been minimized.

Copy link
Contributor

commented Jan 5, 2015

On 6 January 2015 at 05:19, Brad Fitzpatrick notifications@github.com
wrote:

@c4milo https://github.com/c4milo, yeah, root works. I can drop down to
a less-privileged user if I need to. But some of our net tests require
root, or at least more capabilities.

Really? None of the builders I have configured have executed the tests as
root, and I don't think we should be doing that.

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Jan 5, 2015

They skip the tests when they're not root.

I don't think it's a big deal, and let's not get sidetracked on this bug.

@minux

This comment has been minimized.

Copy link
Member

commented Jan 5, 2015

I think at least for 10.6, you are not allowed to run regular OS X inside
VMWare Fusion.
You must use OS X Server.

It's probably also true for 10.7.

PS: we do need a builder for 10.6, because 10.6 is the earliest version
that we still
support.

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Jan 5, 2015

We can do an old-style builder for 10.6, for as long as we still support it, which probably won't be much longer. It's possible we even drop 10.6 support in Go 1.5.

@minux

This comment has been minimized.

Copy link
Member

commented Jan 5, 2015

Removing OS X 10.6 support in 1.5 SGTM.
It's causing too much trouble in support, and its support is crippled by the
fact that external linking is not working anyway.

bradfitz added a commit to golang/tools that referenced this issue Jan 8, 2015
dashboard/buildlet: darwin support
Update golang/go#9495

Change-Id: I732cfdee952ad3bf0b3411d0ce45723900acedb4
Reviewed-on: https://go-review.googlesource.com/2472
Reviewed-by: Andrew Gerrand <adg@golang.org>
@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Jan 13, 2015

Moving from email thread, I confirmed I can boot into Yosemite into a desktop running Terminal and the buildlet in under 1 minute. It's about 57 seconds on my iMac 5K i7 and VMWare Fusion Pro.

The buildlet is accessible on port 8080 from the host, if you know its IP:

buildlet

And you can create a start-up item of a +x file named start-buildlet.command (the *.command extension is key) which launches a shell script in Terminal:

dot-command

We can then snapshot that and create VMs which boot like that, in ~57 seconds. The host API server on the outside should probably cache the ~6MB buildlet too, to reduce network I/O. We can do that later, though.

@c4milo

This comment has been minimized.

Copy link
Member

commented Jan 13, 2015

Here is the first draft of the API: https://github.com/c4milo/osx-builder.

@bradfitz, keep in mind that Login Items will only run upon the user logging in. It might be more convenient for the API if they run on boot as there won't be need to log into the guest OS in order to make the script run. As I mentioned in our email thread, this can be done through Launchd. There is more information about Launchd here: https://developer.apple.com/library/mac/documentation/MacOSX/Conceptual/BPSystemStartup/BPSystemStartup.pdf

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Jan 13, 2015

I configured the user to automatically log in, so the Login Items works great.

@c4milo

This comment has been minimized.

Copy link
Member

commented Jan 13, 2015

oh, yeah that works too. nice!

bradfitz added a commit to golang/build that referenced this issue Jan 21, 2015
dashboard/buildlet: darwin support
Update golang/go#9495

Change-Id: I732cfdee952ad3bf0b3411d0ce45723900acedb4
Reviewed-on: https://go-review.googlesource.com/2472
Reviewed-by: Andrew Gerrand <adg@golang.org>
@c4milo

This comment has been minimized.

Copy link
Member

commented Feb 3, 2015

@bradfitz what is left here?

@adg

This comment has been minimized.

Copy link
Contributor

commented Feb 5, 2015

@c4milo we're going to set up a small cluster of OS X machines to use as builders (@crawshaw is doing that). Once they're set up, we'll know what else needs to be done, if anything.

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Feb 5, 2015

@c4milo in addition to what Andrew said, I also haven't had found time yet to use what you've made. I created a VM in VMWare that boots right into a buildlet and I could hit it from the outside, which was enough to convince me that the concept would work, but I haven't tested it end-to-end.

We'll also need to add support for the daemon that runs on the host to (your code) to connect to the build coordinator and setup a reverse tunnel back to it, for our farm of Macs running in an isolated network.

bradfitz added a commit to golang/build that referenced this issue Mar 31, 2015
cmd/coordinator: add BuildletPool abstraction; don't assume GCE
This starts the process of making the coordinator about to use the
buildlet on things other than GCE.

Update golang/go#8647 (ARM, reverse proxy or online.net)
Update golang/go#10267 (windows 2003 on AWS)
Update golang/go#9495 (OS X VMWare VMs on racked Mac Minis)

Change-Id: I5df79ea67e0ececba8b880e81bd93d4c80897455
Reviewed-on: https://go-review.googlesource.com/8198
Reviewed-by: David Crawshaw <crawshaw@golang.org>

@rsc rsc added this to the Unplanned milestone Apr 10, 2015

@rsc rsc changed the title builders: tracking bug for OS X virtualization x/build/builders: tracking bug for OS X virtualization Apr 14, 2015

@rsc rsc modified the milestones: Unreleased, Unplanned Apr 14, 2015

@rsc rsc removed the builder label Apr 14, 2015

@rsc rsc added the Builders label Jun 11, 2015

gopherbot pushed a commit to golang/build that referenced this issue Aug 30, 2016
dashboard: add OS X 10.8 and 10.11 reverse builders
Updates golang/go#9495
Updates golang/go#12185

Change-Id: I84548011823334f0876636f8e9685e07238cafc4
Reviewed-on: https://go-review.googlesource.com/28171
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot

This comment has been minimized.

Copy link

commented Aug 31, 2016

CL https://golang.org/cl/28218 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Sep 4, 2016
darwin/macstadium: start of OS X VMWare builders
Updates golang/go#9495

Change-Id: Id01533737c14453aaa1bdccaa3124dd7e30a2465
Reviewed-on: https://go-review.googlesource.com/28218
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot

This comment has been minimized.

Copy link

commented Sep 6, 2016

CL https://golang.org/cl/28572 mentions this issue.

@gopherbot

This comment has been minimized.

Copy link

commented Sep 6, 2016

CL https://golang.org/cl/28570 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Sep 6, 2016
env/darwin/macstadium: more OS X setup notes for VMWare-based builders
Updates golang/go#9495

Change-Id: Id87d401d3f7e6fd464039cff4e58a5eda14424cf
Reviewed-on: https://go-review.googlesource.com/28570
Reviewed-by: Chris Broadfoot <cbro@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Sep 7, 2016
env/darwin/macstadium: keep OS X buildlets looping for now
We'll revert them to snapshot later.

Updates golang/go#9495

Change-Id: I93e67598977484f0f3c7eecab07d76370e85fa1d
Reviewed-on: https://go-review.googlesource.com/28580
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Sep 7, 2016
cmd/buildlet: support auto-config from GO_BUILDER_ENV, for OS X build…
…er VMs

The buildlet now checks the GO_BUILDER_ENV environment variable on
start-up and configures itself automatically if set to a known value.

The only currently-known value is "macstadium_vm", which means that
the buildlet will get its role & build key from the OS X version
(using "sw_vers") and the VMWare VM metadata, respectively.

Updates golang/go#9495

Change-Id: I9651b2d589fed46ff3d087a97acc0ea7171e2b24
Reviewed-on: https://go-review.googlesource.com/28572
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Sep 7, 2016
cmd/coordinator, dashboard: switch to new VMWare-based OS X builders
Ignore the old darwin-{amd64,386}-10_10 builders. Don't give them an
error, but pretend they don't exist.

Also: switch trybots from OS X 10.10 to OS X 10.11, and re-enable
sharding. Let's hope for the best. See golang/go#12979.

This also enables subrepo tests for all OS X versions.

darwin-386-* is currently offline, pending some golang/go#17009

Updates golang/go#9495 (OS X virtualization)

Change-Id: I4d53a79087404b5e8051d1aff0c668a92625f442
Reviewed-on: https://go-review.googlesource.com/28583
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot

This comment has been minimized.

Copy link

commented Sep 7, 2016

CL https://golang.org/cl/28584 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Sep 7, 2016
cmd/makemac: add start of command/package to create OS X builder VMs
Currently it's just a package main. It might split in two later. It
might stop shelling out to govc and use the underlying API directly
too, but it's hairy.

Easier to work like this for now.

Updates golang/go#9495

Change-Id: I0d2e19abcb5114ab7fe2e2c543d14e50897d4cbb
Reviewed-on: https://go-review.googlesource.com/28584
Reviewed-by: Quentin Smith <quentin@golang.org>
@quentinmit

This comment has been minimized.

Copy link
Contributor

commented Sep 29, 2016

@bradfitz Can we closed this bug now? farmer thinks there are a bunch of running darwin buildlets, which I think are in virtualization.

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Sep 30, 2016

The coordinator doesn't yet directly manage the creation of VMs.

The current setup is a transitional hack to get us going where they're VMs, but they're created by a tool which creates them statically once, by hand, and then they just act like normal static machines in reverse mode. But if they die, they never come back to life, and we can't react to load or demand. Right now we only create one OS X 10.8 VM, 3 10.8 VMs, and the rest 10.11. That's pretty arbitrary. If four people wanted OS X 10.8 VMs via gomote, we can't create that right now.

So I want to keep this open until things are better integrated.

@gopherbot

This comment has been minimized.

Copy link

commented Feb 25, 2017

CL https://golang.org/cl/37457 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Feb 25, 2017
cmd/coordinator: clean up reverse buildlet code, export status JSON
Will be used for dynamic creation/destruction of Mac VMs in subsequent CL.

Updates golang/go#9495 (Mac virtualization)
Updates golang/go#15760 (monitoring)

Change-Id: I48b17589b258d5d742bad5a3ddae18de98778149
Reviewed-on: https://go-review.googlesource.com/37457
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot

This comment has been minimized.

Copy link

commented Dec 6, 2017

Change https://golang.org/cl/82355 mentions this issue: cmd/buildlet: always halt Mac VMs after builds, regardless of macOS version

gopherbot pushed a commit to golang/build that referenced this issue Dec 7, 2017
cmd/buildlet: always halt Mac VMs after builds, regardless of macOS v…
…ersion

We used to do this only by necessity on Sierra, due to Sierra issues,
but now that x/build/cmd/makemac is better, we can just do this all
the time. Doing it always is better anyway, to guarantee fresh
environments per build.

Also add the forgotten sudo in the Mac halt. The
env/darwin/macstadium/image-setup-notes.txt even calls out how to
setup password-free sudo to the images, but then I forgot to use it.
It only worked before (if it did?) because the process ended and
failed its heartbeat, at least some of the time. It's also possible it
was never working. The old reason that Sierra machines were
special-cased to reboot was reportedly fixed anyway, in
golang/go#18751 (comment).

Updates golang/go#9495

Change-Id: Iea21d7bc07467429cde79f4212c2b91458f8d8d8
Reviewed-on: https://go-review.googlesource.com/82355
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

@bradfitz bradfitz changed the title x/build/builders: tracking bug for OS X virtualization x/build/builders: tracking bug for macOS virtualization Nov 30, 2018

@bradfitz bradfitz added the OS-Darwin label Nov 30, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.