x/build: Mac VMs down

[bradfitz]

I'm on vacation for 2 weeks, but I peeked at why the Mac builders are dead and trybots are thus hanging.

As background, the Macs builders are 10 physical Mac Minis hosted at MacStadium.com, running a VMware vSphere cluster, where we run up to 20 virtual macs on those 10 physical macs.

The Linux box & Go daemon that runs the show is a Linux VM on the same vSphere cluster. Its host name is macstadiumd.golang.org. It's not responding to pings.

My guess is MacStadium updated our vSphere cluster for us for the Spectre/Meltdown patches, which requires a full reboot of all VMware hosts & VMs. (I just had to do this myself for my home vSphere setup)

And I further suspect that our macstadiumd.golang.org VM was never marked to automatically boot on vSphere power-up.

(A previous bug, #20224, was about making sure the daemons on that machine were in systemd at least so they started automatically upon boot, but I guess we forgot to make sure that the machine itself powers on.)

@rsc has the credentials & VPN instructions for getting into the vSphere admin web UI. (be sure to enable Flash in Chrome Content Settings!)

/cc @andybons @rsc @ianlancetaylor

[rsc]

The Linux system had a blank console unresponsive to input. I powered off and powered on the Linux system. Now it has a login prompt, and the Mac VMs started disappearing and reappearing, I assume under the Linux system's management.

[rsc]

The Jan 14 commit on the main dashboard has filled in all the darwin holes it had. A new darwin column for 10.8 has appeared as well - it has just one ok in it now, but I assume that's going to get filled in more as time goes on, or at least is different from this machine being hung. Going to assume the trybots are unstuck now too and close this.