Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/makemac: automatically power cycle dead MacStadium VMware hosts #32033

Closed
bradfitz opened this issue May 14, 2019 · 1 comment
Closed
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@bradfitz
Copy link
Contributor

Our MacStadium VMware host machines have a tendency to die or somehow wedge and stop reporting to vSphere.

MacStadium has an API now to hard power cycle machines:

https://www.macstadium.com/features/api

Sever Action (Power) by IP
URI: https://api.macstadium.com/core/api/servers/{id or ip_address}/{action}
Request: GET
Params: Action can be ‘On’,’Off’, or ‘Reboot’ (case insensitive).
Return: String. ‘True’ on success or an error message.
Example:

curl -H "Accept: application/json" -H "Content-Type: application/json" -X GET -u user:pass https://api.macstadium.com/core/api/servers/192.168.1.101/reboot

We should make our x/build/cmd/makemac daemon monitor the healthy status of our 10 VMware nodes (using the govc API or just whether they've registered with the coordinator in the past N minutes) and if we suspect a node is dead, we hard-reboot it with the API.

/cc @andybons @dmitshur @bcmills

@bradfitz bradfitz added the NeedsFix The path to resolution is known, but the work has not been done. label May 14, 2019
@gopherbot gopherbot added this to the Unreleased milestone May 14, 2019
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label May 14, 2019
@dmitshur
Copy link
Contributor

This issue is obsolete as of #57562. Closing.

@dmitshur dmitshur closed this as not planned Won't fix, can't repro, duplicate, stale May 15, 2023
@golang golang locked and limited conversation to collaborators May 14, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
Archived in project
Development

No branches or pull requests

3 participants