Xcode install on macOS 13 sometimes fails and then continuously fails on following runs

[vashworth]

Is there an existing issue for this?

• I have searched the existing infra issues

Type of Request

bug

Infrastructure Environment

LUCI

What is happening?

Occasionally, Xcode errors to install and then on following runs it will time out when trying to install

Here's example of where the problem starts:
https://luci-milo.appspot.com/ui/p/flutter/builders/try/Mac%20Engine%20Drone/509375/overview

reason: failed to update package permissions in /Volumes/Work/s/w/ir/cache/osx_sdk/xcode_14e300c/XCode.app for ios

Then in following runs, it times out:
https://luci-milo.appspot.com/ui/p/flutter/builders/try/Mac%20Engine%20Drone/509416/overview
Stuck at

Removing the hidden cipd files if exists to be compliant with MacOS13+ codesign check...

Clearing the cache seems to fix.

Steps to reproduce

No response

Expected results

No response

[godofredoc]

This seems like a mac_toolchain issue trying to change the mode of hidden files and then trying to delete those files on subsequent builds. Is there a new version of mac_toolchain that we can update to?

[vashworth]

This seems like a mac_toolchain issue trying to change the mode of hidden files and then trying to delete those files on subsequent builds. Is there a new version of mac_toolchain that we can update to?

So yes, there is newer version of mac_toolchain. However, newer version increases every build by ~2 minutes so we opted to use an older version for now. Also looking at their change history, I wonder if that's what this change is meant to fix.

Maybe as a stopgap, we can update recipes so that if the install xcode steps fails, then delete the cache

[vashworth]

Another example that was using newest version of toolchain:

When it failed to install: https://luci-milo.appspot.com/ui/p/flutter/builders/try/Mac%20build_tests_2_4/54714/overview
The next run it fails integrity check and attempts to reinstall but fails: https://luci-milo.appspot.com/ui/p/flutter/builders/try/Mac%20Engine%20Drone/501115/overview
The next run it passes integrity check and times out: https://luci-milo.appspot.com/ui/p/flutter/builders/try/Mac%20dart_plugin_registry_test/29812/overview

Looking at this example, I don't think newer mac_toolchain will fix.

[vashworth]

We could try a workaround solution to hopefully prevent it from failing on following runs: https://flutter-review.googlesource.com/c/recipes/+/52421

I think we'll need to file an issue with chromium about it failing to update package permissions

[godofredoc]

It seems like https://chromium.googlesource.com/infra/infra/+/32d81d877ee07af07bf03b7f70ce597e323b80ce is fixing the issue. I'd recommend to use the latest version 2m increase in execution time is not desirable but it can potentially fix this P0

[vashworth]

It seems like https://chromium.googlesource.com/infra/infra/+/32d81d877ee07af07bf03b7f70ce597e323b80ce is fixing the issue. I'd recommend to use the latest version 2m increase in execution time is not desirable but it can potentially fix this P0

See this comment that shows newest mac_toolchain does not fix issue: #138238 (comment)

[keyonghan]

Filed https://crbug.com/1501452.

[ricardoamador]

Lowering to P1 as there is a work around and the required fix is being investigated by the Chromium team.

[vashworth]

FYI I believe this is only an issue on chromium bots - I have not seen it happen on devicelab bots

[vashworth]

There seems to be a pattern for when this issue happens:

• A build is cancelled (Internal Failure is true on Swarming Task page), which invalidates the cache
• Next build starts with empty cache (flutter_xcode should not be listed in caches list on swarming task page) and redownloads/installs Xcode
• During install permission error happens and build fails
• During next build, install hangs

[keyonghan]

The workaround to auto retry xcode install is in. Let's monitor if it mitigates the issue.

[vashworth]

The workaround @keyonghan mentioned was https://flutter-review.googlesource.com/c/recipes/+/52421

We found that this wasn't entirely sufficient, though, because if the Xcode version was already corrupted, it would stay corrupted (see #139152). So we added checks to see if it was corrupted first and delete it if so: https://flutter-review.googlesource.com/c/recipes/+/52702

[vashworth]

I've been observing the macOS 13 bots and haven't seen this issue again. I think it's safe to close.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.