Windows builds are 1h+ queue time

[zanderso]

It's 2pm PT on a Tuesday. There are 14 busy and 46 idle prod Windows bots, and there are 96 busy and 0 idle try Windows bots. The engine -> framework roll Windows presubs have been queueing for >1hr.

Do we have the right prod/try balance for peak times? Can the idle prod bots lend a hand to the try bots?

cc @keyonghan @godofredoc

[godofredoc]

Unfortunately bots are statically assigned per pool. There is a possibility to use dynamic allocation but it requires a couple of migrations to use the functionality.

[godofredoc]

@yusuf-goog FYI

[ditman]

We may have too many concurrent PRs open, but PRs in flutter/packages have been stalled for >1h to start (and still waiting), for example:

• https://github.com/flutter/packages/pull/3635/checks?check_run_id=12521438058
• https://github.com/flutter/packages/pull/3601/checks?check_run_id=12521283818

This "scarcity" (?) seems to have started in the last couple of hours or so?

[keyonghan]

https://flutter-review.googlesource.com/c/recipes/+/41460 to use same priority for both drone build and main build, validating how the higher priority of drone builds affects the increased queue time in regular builds.

[keyonghan]

Some context:

• Windows host only queue time increased to 7 min around 03/15
• Another bump happened around 03/27 to 40-50 mins, which aligned with some changes to enable 16 more shards (flutter/engine@bcd6d3c)
• Fail the run_suite_step if any test failures are detected. engine#40834 landed yesterday to lower the shard number from 16 to 8
• Windows host only executing time is consistent for months, ~26min (90th%)
• Linux and Mac platforms are not affected

[keyonghan]

Tracking bug to improve the flake bot: #124272
More capacity cl: https://chrome-internal-review.googlesource.com/c/infradata/config/+/5711329

[ditman]

We've experienced much, much better performance than yesterday in the flutter/packages repo!

[godofredoc]

Thanks @ditman for the information, we'll continue with the research and fixing the issues we have found so far. A few things not mentioned in this bug yet:

• Closed builds not releasing running/pending tasks
• The queue time metric needs to be updated as the current percentile may be filtering out long queue times if they happen for less than 2 hours per day.

[keyonghan]

As of now, linux queue time drops <5min, windows 0 pending tasks, mac <10mins, this is consistent with our bi-weekly data. Issue has been mitigated, and some tracking bugs:

1. Re-evaluate how to create flake/deflake PRs (Re-evaluate how to create flake/deflake PRs #124272) to avoid 10 PRs created around the same time
2. cancel scheduled/running tasks when closing PRs (Running builds are not being stopped in LUCI when a PR is closed #124238)
3. Track 99th% queue time to catch short term peaks of queue time
4. Identify our cpu quota to allocate more available linux/windows vm capacity @yusuf-goog Please share a tracking bug if any.

Closing here with remaining issues tracked separately.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.