Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: all darwin builders are missing #40929

Closed
cagedmantis opened this issue Aug 20, 2020 · 3 comments
Closed

x/build: all darwin builders are missing #40929

cagedmantis opened this issue Aug 20, 2020 · 3 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Soon This needs action soon. (recent regressions, service outages, unusual time-sensitive situations)
Milestone
Unreleased

Comments

@cagedmantis
Copy link
Contributor 

# Notes: https://github.com/golang/build/tree/master/env/darwin/macstadium
Warn: macstadium_host09a missing, not seen for 13h21m50s
Warn: macstadium_host09b missing, not seen for 13h26m46s
Warn: 2 machines missing, 10% of capacity
Warn: makemac daemon: vm.destroy("mac_10_12_host09b") = govc vm.destroy ...: exit status 1, govc: Unable to communicate with the remote host, since it is disconnected.

Warn: makemac daemon: vm.destroy("mac_10_11_host09a") = govc vm.destroy ...: exit status 1, govc: Unable to communicate with the remote host, since it is disconnected.

Similar recent issues: #39378, #39306

/cc @andybons @dmitshur @toothrot
@cagedmantis cagedmantis added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Aug 20, 2020
@cagedmantis cagedmantis added this to the Unreleased milestone Aug 20, 2020
@cagedmantis cagedmantis self-assigned this Aug 20, 2020
@cagedmantis
Copy link
Contributor Author

MacStadium is investigating

# Notes: https://github.com/golang/build/tree/master/env/darwin/macstadium
Warn: macstadium_host01a missing, not seen for 4m44s
Warn: macstadium_host01b missing, not seen for 4m46s
Warn: macstadium_host02a missing, not seen for 4m39s
Warn: macstadium_host03a missing, not seen for 4m14s
Warn: macstadium_host04a missing, not seen for 3m45s
Warn: macstadium_host05a missing, not seen for 4m18s
Warn: macstadium_host06b missing, not seen for 30m28s
Warn: macstadium_host09a missing, not seen for 14h5m7s
Warn: macstadium_host09b missing, not seen for 14h10m3s
Warn: macstadium_host10b missing, not seen for 2m50s
Error: 10 machines missing, 50% of capacity
Error: makemac daemon: failed to fetch status: Get http://macstadiumd.golang.org:8713: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)```

@cagedmantis
Copy link
Contributor Author

MacStadium is continuing to investigate.

# Notes: https://github.com/golang/build/tree/master/env/darwin/macstadium
Warn: macstadium_host01a missing, not seen for 1h20m6s
Warn: macstadium_host01b missing, not seen for 1h20m7s
Warn: macstadium_host02a missing, not seen for 1h20m1s
Warn: macstadium_host02b missing, not seen for 52m47s
Warn: macstadium_host03a missing, not seen for 1h19m35s
Warn: macstadium_host03b missing, not seen for 1h9m55s
Warn: macstadium_host04a missing, not seen for 1h19m6s
Warn: macstadium_host04b missing, not seen for 1h4m14s
Warn: macstadium_host05a missing, not seen for 1h19m39s
Warn: macstadium_host05b missing, not seen for 24m17s
Warn: macstadium_host06a missing, not seen for 32m50s
Warn: macstadium_host06b missing, not seen for 1h45m50s
Warn: macstadium_host07a missing, not seen for 25m54s
Warn: macstadium_host07b missing, not seen for 1h10m6s
Warn: macstadium_host08a missing, not seen for 1h10m40s
Warn: macstadium_host08b missing, not seen for 14m37s
Warn: macstadium_host09a missing, not seen for 15h20m28s
Warn: macstadium_host09b missing, not seen for 15h25m25s
Warn: macstadium_host10a missing, not seen for 14m48s
Warn: macstadium_host10b missing, not seen for 1h18m11s
Error: 20 machines missing, 100% of capacity
Error: makemac daemon: failed to fetch status: Get http://macstadiumd.golang.org:8713: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)```

@dmitshur dmitshur added the Soon This needs action soon. (recent regressions, service outages, unusual time-sensitive situations) label Aug 20, 2020
@dmitshur dmitshur pinned this issue Aug 20, 2020
@dmitshur dmitshur changed the title x/build: two darwin builders are missing x/build: all darwin builders are missing Aug 20, 2020
@cagedmantis
Copy link
Contributor Author

We've been able to VPN into the network. The bastion host has been started up. Builds have resumed.

ok

@cagedmantis cagedmantis unpinned this issue Aug 20, 2020
@golang golang locked and limited conversation to collaborators Aug 20, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Soon This needs action soon. (recent regressions, service outages, unusual time-sensitive situations)
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
3 participants
@cagedmantis @dmitshur @gopherbot