Mac M1 arm64 runners not available

[huydhn]

Current Status

ongoing

Error looks like

All M1 mac arm64 tests are the queue https://hud.pytorch.org/metrics waiting for macos-m1-12 runners. For example https://github.com/pytorch/pytorch/runs/8025452271

Incident timeline (all times pacific)

The issue started from around 10AM in the morning according to HUD https://hud.pytorch.org. Here is the first commit in trunk that fails due to time out https://hud.pytorch.org/pytorch/pytorch/commit/86e134ddf777d6f4b82a1860102698ef13bf0c33

User impact

Mac M1 arm64 trunk tests waits till timeout. If people have ciflow/trunk in the PR, that would also be impacted.

Root cause

TBD

Mitigation

TBD

Prevention/followups

TBD

[huydhn]

Thank @seemethere for picking up the investigation!

[malfet]

Wow, looks like we are clairvoyant: 19229d days ago is probably Jan 1st 1970

[seemethere]

Looks like an auto-update made it so that the GHA daemon was endlessly boot looping. Seems to coincide with the daemons updating.

I've submitted a fix and re-kicked the daemon on all of the hosts.

marking this a resolved

[ZainRizvi]

@seemethere any plans for RelEng to add alerting to detect these kind of failures faster in the future?