New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: Mac VMs down #23441

Closed
bradfitz opened this Issue Jan 14, 2018 · 2 comments

Comments

Projects
None yet
3 participants
@bradfitz
Member

bradfitz commented Jan 14, 2018

I'm on vacation for 2 weeks, but I peeked at why the Mac builders are dead and trybots are thus hanging.

As background, the Macs builders are 10 physical Mac Minis hosted at MacStadium.com, running a VMware vSphere cluster, where we run up to 20 virtual macs on those 10 physical macs.

The Linux box & Go daemon that runs the show is a Linux VM on the same vSphere cluster. Its host name is macstadiumd.golang.org. It's not responding to pings.

My guess is MacStadium updated our vSphere cluster for us for the Spectre/Meltdown patches, which requires a full reboot of all VMware hosts & VMs. (I just had to do this myself for my home vSphere setup)

And I further suspect that our macstadiumd.golang.org VM was never marked to automatically boot on vSphere power-up.

(A previous bug, #20224, was about making sure the daemons on that machine were in systemd at least so they started automatically upon boot, but I guess we forgot to make sure that the machine itself powers on.)

@rsc has the credentials & VPN instructions for getting into the vSphere admin web UI. (be sure to enable Flash in Chrome Content Settings!)

/cc @andybons @rsc @ianlancetaylor

@gopherbot gopherbot added this to the Unreleased milestone Jan 14, 2018

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Jan 15, 2018

Contributor

The Linux system had a blank console unresponsive to input. I powered off and powered on the Linux system. Now it has a login prompt, and the Mac VMs started disappearing and reappearing, I assume under the Linux system's management.

Contributor

rsc commented Jan 15, 2018

The Linux system had a blank console unresponsive to input. I powered off and powered on the Linux system. Now it has a login prompt, and the Mac VMs started disappearing and reappearing, I assume under the Linux system's management.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Jan 15, 2018

Contributor

The Jan 14 commit on the main dashboard has filled in all the darwin holes it had. A new darwin column for 10.8 has appeared as well - it has just one ok in it now, but I assume that's going to get filled in more as time goes on, or at least is different from this machine being hung. Going to assume the trybots are unstuck now too and close this.

Contributor

rsc commented Jan 15, 2018

The Jan 14 commit on the main dashboard has filled in all the darwin holes it had. A new darwin column for 10.8 has appeared as well - it has just one ok in it now, but I assume that's going to get filled in more as time goes on, or at least is different from this machine being hung. Going to assume the trybots are unstuck now too and close this.

@rsc rsc closed this Jan 15, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment