Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: Mac VMs down #23441

Closed
bradfitz opened this issue Jan 14, 2018 · 2 comments

Comments

Projects
None yet
3 participants
@bradfitz
Copy link
Member

commented Jan 14, 2018

I'm on vacation for 2 weeks, but I peeked at why the Mac builders are dead and trybots are thus hanging.

As background, the Macs builders are 10 physical Mac Minis hosted at MacStadium.com, running a VMware vSphere cluster, where we run up to 20 virtual macs on those 10 physical macs.

The Linux box & Go daemon that runs the show is a Linux VM on the same vSphere cluster. Its host name is macstadiumd.golang.org. It's not responding to pings.

My guess is MacStadium updated our vSphere cluster for us for the Spectre/Meltdown patches, which requires a full reboot of all VMware hosts & VMs. (I just had to do this myself for my home vSphere setup)

And I further suspect that our macstadiumd.golang.org VM was never marked to automatically boot on vSphere power-up.

(A previous bug, #20224, was about making sure the daemons on that machine were in systemd at least so they started automatically upon boot, but I guess we forgot to make sure that the machine itself powers on.)

@rsc has the credentials & VPN instructions for getting into the vSphere admin web UI. (be sure to enable Flash in Chrome Content Settings!)

/cc @andybons @rsc @ianlancetaylor

@gopherbot gopherbot added this to the Unreleased milestone Jan 14, 2018

@rsc

This comment has been minimized.

Copy link
Contributor

commented Jan 15, 2018

The Linux system had a blank console unresponsive to input. I powered off and powered on the Linux system. Now it has a login prompt, and the Mac VMs started disappearing and reappearing, I assume under the Linux system's management.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Jan 15, 2018

The Jan 14 commit on the main dashboard has filled in all the darwin holes it had. A new darwin column for 10.8 has appeared as well - it has just one ok in it now, but I assume that's going to get filled in more as time goes on, or at least is different from this machine being hung. Going to assume the trybots are unstuck now too and close this.

@rsc rsc closed this Jan 15, 2018

@golang golang locked and limited conversation to collaborators Jan 15, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.