Large (5.6hrs) linux.gcp.a100.large queue time

[atalman]

Current Status

ongoing

Error looks like

Jobs using linux.gcp.a100.large workers wait long time in queue

Incident timeline (all times pacific)

Jul 12 - 7:00AM Queue time of 2.9 Hrs for linux.gcp.a100.large
Jul 12 - 9:00AM Started investigating as part of OSS CI oncall

User impact

Jobs using linux.gcp.a100.large workers wait long time in queue

Root cause

to be determined

Mitigation

to be determined

Prevention/followups

to be determined

[jeanschmidt]

Closing the issue, as it seems not to be triggered by a lack of available instances, but instead of overload of usage triggered by some users.

[jeanschmidt]

Removed the label in order to avoid confounding when we extract statistics in relationship to our CI outages.