Xcode cache mysteriously gets cleared on macOS #138246
Comments
vashworth
added
team-infra
|
I think this is a pretty big issue and could use infra help |
|
Related issue: #138238 |
vashworth
changed the title
|
Considering these three tasks:
Something went wrong with 2), after the task was canceled, the xcode cache dir (osx_sdk) doesn't seem to get handled properly. This is based on the bot_cache list in task 3) where no Wondering if our recipes module has some special logic. If not then something to check with LUCI team. |
|
Don't we have logic to prevent against a corrupted cache? Looks like it was killed during the download phase. Wait, is this the new method of saving package for mac OS13 @vashworth we spoke about yesterday? |
I don't think it's relevant to the "download dependencies" step if that's what you mean - that steps downloads dart dependencies I think. We have logic against corrupted cache when runtimes are involved, but we would see steps for clearing the cache if that was the case. The only cache clearing logic I'm aware of is when we do it through steps (which none of the builds have) |
|
Started a thread in LUCI. Let's see what they find. |
|
FYI I don't think is specific to macOS 13 - I believe this also happens on macOS 12. This may have been going on for a while... Example on macOS 12: I think this issue is just more exacerbated on macOS 13 because of #138238 |
vashworth
changed the title
|
Confirmed from LUCI team, there is a bug mishandling (or rather not handling at all) of SIGTERM in run_isolated when it is doing cleanup. Will paste the bug when available. |
|
I wonder what fixing this will have an effect on? How much will this help? |
|
Reading more carefully this is the expected behavior for caches. If a download fails in the middle then the cache is corrupted and needs to be downloaded the next time. Rather than focusing on the cache we may need to focus on the failures. |
It's not always the download that fails in the middle (see 2nd example in original description of this issue). But yeah we should focus on the failure |
So this does not seem like an infra issue now based on what LUCI has said. We have a passing test followed by two killed tests. Which then LUCI is not handling the error code properly but the cache is doing what it needs to do. I also have questions as to the first three tests. The first test succeeded, the second was killed in the middle of the download and the third was killed again during the reset of xcode. This doesn't seem like a caching issue. Adding LUCI bug here: https://bugs.chromium.org/p/chromium/issues/detail?id=1501443 |
There's an [issue](flutter/flutter#138238) with mac_toolchain that makes Xcode installs flakey and an [issue](flutter/flutter#138246) that makes Xcode installs more frequent on macOS 13, which is causing presubmit tests to fall frequently. In the meantime, we'll only have tests run on macOS 12. [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style
There's an flutter/flutter#138238 with mac_toolchain that makes Xcode installs flakey and an flutter/flutter#138246 that makes Xcode installs more frequent on macOS 13, which is causing presubmit tests to fall frequently. In the meantime, we'll only have tests run on macOS 12.
|
Lowering to P1 as the issue is off to LUCI to investigate. |
ricardoamador
added
P1
|
Lowing to P2 for LUCI fix. If we see this issue more frequently, let's bump the priority of the LUCI bug. |
There's an flutter#138238 with mac_toolchain that makes Xcode installs flakey and an flutter#138246 that makes Xcode installs more frequent on macOS 13, which is causing presubmit tests to fall frequently. In the meantime, we'll only have tests run on macOS 12.
There's an flutter/flutter#138238 with mac_toolchain that makes Xcode installs flakey and an flutter/flutter#138246 that makes Xcode installs more frequent on macOS 13, which is causing presubmit tests to fall frequently. In the meantime, we'll only have tests run on macOS 12.
flutter-triage-bot
bot
added
the
⏳
|
This issue is assigned to @ricardoamador and @keyonghan but has had no recent status updates. Please consider unassigning this issue if it is not going to be addressed in the near future. This allows people to have a clearer picture of what work is actually planned. Thanks! |
|
FYI this does still appear to be a problem, but it's not as critical since when re-installing, it only re-installs Xcode since the simulator was still mounted (takes ~2 minutes rather than ~10). For example, |
flutter-triage-bot
bot
removed
the
⏳
Is there an existing issue for this?
Type of Request
bug
Infrastructure Environment
LUCI
What is happening?
In one build Xcode is downloaded: https://chromium-swarm.appspot.com/task?id=65cf2793c5a1a110
Next build was canceled: https://chromium-swarm.appspot.com/task?id=65cf4e5b7efc7b10
In next build the cache is suddenly empty (see step "Show xcode cache") and Xcode has to be downloaded again and fails: https://chromium-swarm.appspot.com/task?id=65cf57470c8f4c10
It seems like maybe anytime a build is cancelled in the middle, the Xcode cache is cleared
Here's another example:
Build canceled: https://luci-milo.appspot.com/ui/p/flutter/builders/try/Mac%20tool_integration_tests_2_4/29778/overview
Next build the cache is empty and Xcode fails to re-download: https://luci-milo.appspot.com/ui/p/flutter/builders/try/Mac_arm64%20macos_platform_tests%20master%20-%20packages/8150/overview
Steps to reproduce
No response
Expected results
I expect Xcode cache not to get deleted and for Xcode not to fail to install
The text was updated successfully, but these errors were encountered: