Potential macOS CI setup issue

[keith]

rules_apple tests started failing without any changes there between

this successful build https://buildkite.com/bazel/rules-apple-darwin/builds/4659 on 10/20
and this failing build https://buildkite.com/bazel/rules-apple-darwin/builds/4664 on 10/22

This type of test failure:

		The test runner encountered an error (Failed to establish communication with the test runner. If you believe this error represents a bug, please attach the result bundle at /Users/buildkite/Library/Developer/Xcode/DerivedData/temporary-hgmhctgbjdblljdswqfgutoosvlw/Logs/Test/Test-Transient Testing-2021.10.29_17-57-24-+0000.xcresult. (Underlying Error: Couldn’t communicate with a helper application. Try your operation again. If that fails, quit and relaunch the application and try again. The connection to service on pid 0 named com.apple.testmanagerd.control was invalidated.))

Is often related to the machine setup, but it's hard to say specifically what would be the issue here if the Xcode -runFirstLaunch command was run after any changes.

[thii]

@philwo Can you take a look?

[philwo]

Hi @keith!

I checked the machines again, I found a little issue (three of them were missing the $HOME/Library/Developer directory, which caused issues in the past) and fixed it, but the test still fails with the same issue.

The machines were upgraded to macOS Big Sur and Xcode 13.0 during that window and then I regenerated the buildkite user's home directory from scratch.. it's hard to say what that might have caused :/

Do you have any idea how to debug this? Does the test work fine on a similarly setup developer machine and only fail on Bazel CI?

[thii]

@philwo The test works locally for me with the same Xcode version.

The error looks pretty similar to this one: https://stackoverflow.com/a/67699037/2780476.

You can reproduce it by running the ButtonsMacLogicTests which was disabled in rules_apple: https://github.com/bazelbuild/rules_apple/pull/1264/files

[keith]

If it is that this case is potentially not auto-logging in the user? We do that on our CI and then immediately lock the screen

[thii]

@meteorcloudy Pinging you since looks like you are the owner here now. This issue has been blocking us from using BazelCI for Tulsi for a while now.

This is an infra issue, that started happening after an Xcode upgrade, so it would be nice if it could be revisited in the next Xcode upgrade.

[meteorcloudy]

OK, I'll look into this. Can you rebase bazelbuild/rules_apple#1264 and check if the error persist? I reran the CI presubmit job, it's now failing with a different error.

[thii]

Rebased!

[meteorcloudy]

OK, looks like it's still an issue.

If it is that this case is potentially not auto-logging in the user? We do that on our CI and then immediately lock the screen

@keith How did you do that on your CI? Maybe we can check if we can/should do the same on Bazel CI.
/cc @fweikert

[philwo]

We are auto-logging in a user after boot (it wouldn't hurt to verify that this still works, of course), but the user who is logged in (ci) is not the same user that the Buildkite jobs run as (buildkite). Maybe that's the issue?

[keith]

worth a try

[meteorcloudy]

@philwo Are you going to try it? Or can you point the script that does the auto-logging in?

[philwo]

I don't think there's anything we can try here. It's not possible to login as the buildkite user, because the entire user is wiped and all of its processes are killed after each job. :/

The auto-login is just a system setting that can be modified via System Preferences in "Users & Groups" and then "Login Options".

You can also try to do it on the CLI via this third-party tool (https://github.com/xfreebird/kcpassword), but it hasn't been updated in 7 years, I'm not sure if it still works:

sudo -H -u ci brew install xfreebird/utils/kcpassword
enable_autologin "$USERNAME" "$PASSWORD"

[meteorcloudy]

Oh, I see. We are running the CI jobs as a one-time disposable user, so it must be different from the user for auto-logging in.

[meteorcloudy]

I guess there is little we can do except upgrading Xcode and hope this issue can go away.

[keith]

I rebased the linked PR and do still see the issue. I also found this related post https://stackoverflow.com/questions/67688130/run-macos-test-cases-on-the-jenkins-pipeline but I doubt the buildkite setup has this same issue. It does imply logins might be the issue.

If you're hoping Xcode updates fix things, 13.3 is available now, if you could update the machines that would be great in general

[meteorcloudy]

If you're hoping Xcode updates fix things, 13.3 is available now, if you could update the machines that would be great in general

We need some time to prepare the upgrade as Xcode 13.3 removed python2 support, some things are going to break..

[keith]

If the machines are already on 12.X you don't have to update to 12.3 and drop python2 for Xcode 13.3, but if they are still on 11.x you might have to go straight to the newest (unless you're using MDM and can block that point release)

[meteorcloudy]

Thanks for the advice, I just checked, we are still on macOS 11.6.1, so it could be a bit tricky.

[keith]

@meteorcloudy any plans to address this soon?

[meteorcloudy]

Sorry, this is still on our list, but we currently don't have capacity to look into it.