Runtime jobs being abandoned due to infra #50746
Comments
|
I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label. |
dotnet-issue-labeler
bot
added
the
untriaged
jkoritzinsky
added
blocking-clean-ci
|
Tagging subscribers to this area: @dotnet/runtime-infrastructure Issue DetailsRunfo Tracking Issue: Runtime jobs being abandoned due to infra
Build Result Summary
|
|
This has started popping up again. @dotent/dnceng |
|
cc: @adiaaida this is the same issue that I shared on the FR channel. Happening not only on windows. |
|
Got it. Thanks. @dotnet/dnceng I have opened https://github.com/dotnet/core-eng/issues/12732 to track this on our side |
|
@safern These all appear to be running on windows. Is that incorrect? |
|
Sorry I miss read somehow, yes they are all windows 🤦 |
|
adding @lukas-lansky |
|
Let's look!
|
|
It was actually Ulises who suspected the scaler. When we saw this was happening, he immediately manually scaled up the queue, and that's why you see that drop off in wait time. |
|
Hmm, so is the issue supposed to be mitigated? It seems to me that all Windows legs in PR / CI runs are stuck right now, is that just a backlog caused by the previous slowdown or has the problem reappeared even with the upscaled queue? |
|
It appears there was an issue with the underlying service fabric framework that has been mitigated. We trying to manually scale this queue. |
|
Thanks Ilya for the clarification; I have also realized that my previous formulation was kind of selfish, what I meant to say was that "all Windows legs in my PR / CI runs are stuck right now" and that continues to be the case, according to your comment for now I just hope that thanks to your manual adjustments the backlog will eventually disappear unless you advise me to take some proactive measures like abandoning my currently running tests and triggering new ones. |
|
I don't think HelixProd scalesets have been fixed, I keep getting error trying to scale them up. @ilyas1974 @adiaaida we should create an IcM The affected scalesets are buildpool.windows.10.amd64.vs2017.open-a-scaleset and buildpool.windows.10.amd64.vs2019.open-a-scaleset |
|
I've created Azure support ticket TrackingID#2104060010003005 for this issue. |
|
Created a second ticket for vs2019 queue ID#2104070010000075 |
|
The issue is still active, no response from Azure support. I chased 2104070010000075. |
|
Because there is no update I also create ICM ticket https://portal.microsofticm.com/imp/v3/incidents/details/235578196/home |
ViktorHofer
added
tracking-external-issue
Should help mitigate dotnet#50746
* Switch to VS preview pool for public builds Should help mitigate #50746 * Run init-vs-env.cmd for Browser wasm Windows build The BuildPool.Windows.10.Amd64.VS2019.Pre.Open queue doesn't have ninja installed outside of VS so it's only available in PATH if you run the init-vs-env.cmd script.
Reverts dotnet#50993, the underlying Azure issue was fixed. Closes dotnet#50746
ghost
added
the
in-pr
ghost
removed
the
in-pr
ghost
locked as resolved and limited conversation to collaborators
Runfo Tracking Issue: Runtime jobs being abandoned due to infra
Build Result Summary
The text was updated successfully, but these errors were encountered: