Upgrade Chrome on CI builders, "Mac web_tool_tests" timing out on macOS 12

[zanderso]

I misidentified the issue as the Xcode upgrade reverted here #98263.

After the revert that shard still has sub tests timing out.

@yjbanov @annagrin

[annagrin]

@zanderso I wonder if the priority is correct on this - we normally get P2 if the tests cause more than 2% flakes. Do we have data on flakiness for those tests yet?

[annagrin]

@yjbanov this would require investigation to detect a component that caused the flake.
@christopherfujino appreciate your help this.

[zanderso]

Yes, P0 is correct. 7 of the last 10 runs have failed on the tree since yesterday evening, and most framework PRs including rolls are blocked on this. https://ci.chromium.org/p/flutter/builders/prod/Mac%20web_tool_tests

[zanderso]

@godofredoc Aside from the Xcode change, were there any other changes in infra yesterday that might have caused this test to become extremely flaky?

[zanderso]

All pre- and post- submit builds that have picked up #98263 have been passing. /cc @yusuf-goog

[godofredoc]

No we haven't changed anything that will affect this test. The only changes that have been submitted to recipes are engine related changes.

[godofredoc]

Can we try running the test with the old version of xcode in a mac 12 machine?

[christopherfujino]

@yjbanov this would require investigation to detect a component that caused the flake.
@christopherfujino appreciate your help this.

Taking a look now

[christopherfujino]

I misidentified the issue as the Xcode upgrade reverted here #98263.

After the revert that shard still has sub tests timing out.

@yjbanov @annagrin

Actually, it looks to me like that last failure from https://ci.chromium.org/p/flutter/builders/prod/Mac%20web_tool_tests used Xcode 13a233:

• last failing CI build (triggered 09:59 PST Feb 11): https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20web_tool_tests/4898/overview
• install xcode step: https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8822503141826671953/+/u/install_xcode/execution_details

[christopherfujino]

The two newer builds that have run are green (and use the previous xcode version): https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20web_tool_tests/4899/overview and https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20web_tool_tests/4900/overview

[yusuf-goog]

This one passed and used 13a233: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20web_tool_tests/4897/overview

[christopherfujino]

It flaked on post-submit the commit that reverted the Xcode bump #98263 I don't think it's related to #98144 https://ci.chromium.org/p/flutter/builders/prod/Mac%20web_tool_tests/4898.

Zach already said this, please ignore the noise 🙂

After the revert that shard still has sub tests timing out.

The reason it failed on that commit is because this doesn't actually take effect until the next build, since the .ci.yaml is not parsed from the commit, but auto-rolled to the builder config post-submit.

[jmagman]

Sorry for the noise you already figured all that out, I had too aggressively skimmed this issue. I read it more closely and am now up to speed. 🔎

[christopherfujino]

Sorry for the noise you already figured all that out, I had too aggressively skimmed this issue. I read it more closely and am now up to speed. 🔎

No problem, I had missed that it flaked the first time when I commented #98278 (comment). Best to have all this clearly documented on the issue, lest someone else investigate it all again.

[jmagman]

@godofredoc Do we know why this wasn't caught in staging? I thought we ran all the tests against the new macOS and Xcode version.

[godofredoc]

I believe we didn't run all the mac tests on staging we were relying on the presubmit checks.

[christopherfujino]

@yjbanov

1. are we sure that updating Chrome will resolve this issue?
2. are there any updates on fixing this in the framework on LUCI?

[yjbanov]

Locally Chrome 96 runs fine on the latest macOS version. That's the only evidence I have that updating Chrome may resolve this issue. If not, we'll have to investigate further. macOS frequently makes changes that require you to upgrade your apps, so trying to make an ancient version of Chrome to run on the new macOS doesn't seem useful.

[zanderso]

What is the current status of this? In chat I see that this might be currently waiting on flutter/gallery#604 to land? Is that correct?

[ditman]

@zanderso no, I think the gallery update can happen in parallel to this issue. The update to the framework to recognize the new metric names from Chrome 89+ landed here: #98629

[jmagman]

Thanks @ditman. What are next steps then, do the cipd packages have the tags they need for us to flip the version in .ci.yaml?

[ditman]

@jmagman I think @yjbanov had a PR where he attempted to update the version of Chrome via a new CIPD tag, but I have no idea how to recreate that PR. Is it a generated change or does this need to be done manually?

This was the PR: #93289

[jmagman]

Is it a generated change or does this need to be done manually?

Manually. Sounds like the infra team prefers using verson: cipd tags #93289 (comment).

-           {"dependency": "chrome_and_driver", "version": "version:84"}, 
+           {"dependency": "chrome_and_driver", "version": "version:96"}, 

flutter/.ci.yaml

Line 85 in 0eedf42

{"dependency": "chrome_and_driver", "version": "version:84"},

@yjbanov were you working with @hterkelsen on getting adding the tags to https://chrome-infra-packages.appspot.com/p/flutter_internal/browsers/chrome ?

[ditman]

This migration happened here: #99318 (to Chromium 98)

[jmagman]

https://ci.chromium.org/p/flutter/builders/prod/Mac%20web_tool_tests/5087 succeeded after the upgrade to macOS 12 and Chromium 98. Thanks for working on this @yjbanov @ditman @godofredoc!
Closing.

[zanderso]

The Engine roll here #99362 is failing on Mac web_tool_tests: https://ci.chromium.org/ui/p/flutter/builders/try/Mac%20web_tool_tests/12828/overview. Does it need to be rebased?

[jmagman]

I think that machine has the malformed cipd package cached, filed #99372

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.