x/build/cmd/makemac: automatically power cycle dead MacStadium VMware hosts #32033
Labels
Builders
x/build issues (builders, bots, dashboards)
FrozenDueToAge
NeedsFix
The path to resolution is known, but the work has not been done.
Milestone
Our MacStadium VMware host machines have a tendency to die or somehow wedge and stop reporting to vSphere.
MacStadium has an API now to hard power cycle machines:
https://www.macstadium.com/features/api
We should make our x/build/cmd/makemac daemon monitor the healthy status of our 10 VMware nodes (using the
govc
API or just whether they've registered with the coordinator in the past N minutes) and if we suspect a node is dead, we hard-reboot it with the API./cc @andybons @dmitshur @bcmills
The text was updated successfully, but these errors were encountered: