Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.
Sign upx/build: Mac VMs down #23441
Comments
bradfitz
added
Builders
NeedsFix
labels
Jan 14, 2018
bradfitz
assigned
rsc
Jan 14, 2018
gopherbot
added this to the Unreleased milestone
Jan 14, 2018
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
rsc
Jan 15, 2018
Contributor
The Linux system had a blank console unresponsive to input. I powered off and powered on the Linux system. Now it has a login prompt, and the Mac VMs started disappearing and reappearing, I assume under the Linux system's management.
|
The Linux system had a blank console unresponsive to input. I powered off and powered on the Linux system. Now it has a login prompt, and the Mac VMs started disappearing and reappearing, I assume under the Linux system's management. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
rsc
Jan 15, 2018
Contributor
The Jan 14 commit on the main dashboard has filled in all the darwin holes it had. A new darwin column for 10.8 has appeared as well - it has just one ok in it now, but I assume that's going to get filled in more as time goes on, or at least is different from this machine being hung. Going to assume the trybots are unstuck now too and close this.
|
The Jan 14 commit on the main dashboard has filled in all the darwin holes it had. A new darwin column for 10.8 has appeared as well - it has just one ok in it now, but I assume that's going to get filled in more as time goes on, or at least is different from this machine being hung. Going to assume the trybots are unstuck now too and close this. |
bradfitz commentedJan 14, 2018
•
edited
I'm on vacation for 2 weeks, but I peeked at why the Mac builders are dead and trybots are thus hanging.
As background, the Macs builders are 10 physical Mac Minis hosted at MacStadium.com, running a VMware vSphere cluster, where we run up to 20 virtual macs on those 10 physical macs.
The Linux box & Go daemon that runs the show is a Linux VM on the same vSphere cluster. Its host name is
macstadiumd.golang.org. It's not responding to pings.My guess is MacStadium updated our vSphere cluster for us for the Spectre/Meltdown patches, which requires a full reboot of all VMware hosts & VMs. (I just had to do this myself for my home vSphere setup)
And I further suspect that our macstadiumd.golang.org VM was never marked to automatically boot on vSphere power-up.
(A previous bug, #20224, was about making sure the daemons on that machine were in systemd at least so they started automatically upon boot, but I guess we forgot to make sure that the machine itself powers on.)
@rsc has the credentials & VPN instructions for getting into the vSphere admin web UI. (be sure to enable Flash in Chrome Content Settings!)
/cc @andybons @rsc @ianlancetaylor