Mac_arm64 tool_host_cross_arch_tests is failing with recipe crash

[jmagman]

https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8800893589239561057/+/u/RECIPE_CRASH__Uncaught_exception_/logging

The recipe has crashed at point 'Uncaught exception'!

Traceback (most recent call last):
  File "/Volumes/Work/s/w/ir/kitchen-checkout/recipe_engine/recipe_engine/internal/engine.py", line 585, in run_steps
    raw_result = recipe_obj.run_steps(api, engine)
  File "/Volumes/Work/s/w/ir/kitchen-checkout/recipe_engine/recipe_engine/internal/recipe_deps.py", line 936, in run_steps
    recipe_result = invoke_with_properties(
  File "/Volumes/Work/s/w/ir/kitchen-checkout/recipe_engine/recipe_engine/internal/property_invoker.py", line 90, in invoke_with_properties
    return _invoke_with_properties(callable_obj, all_props, environ, prop_defs,

This test is marked bringup:true due to #112130, but was previously successfully passing in staging.

[jmagman]

cc oncall @drewroengoogle

[drewroengoogle]

It seems like the bundler path isn't being recognized at all, but I'm unsure what changed between the time this recipe was passing and failing, as the bundler version seems to be the same, and the recipe hasn't been modified during that timeframe. I can try taking this on.

[drewroengoogle]

Unassigning after discussion with @godofredoc about whether this is an infra issue.

[jmagman]

@drewroengoogle do you know who is working on this? Is /Volumes/Work/s/w/ir/cache/ruby/bin/bundle missing? We want to add more Mac_arm64 tests #110126

[drewroengoogle]

@jmagman Thanks for following up, I meant to yesterday but got sidetracked.
I sent a message in the LUCI chat this morning and CC'ed you, as I'm not sure the best person to take a look at this. From my search it doesn't seem like there was any difference between the last successful run and the first failing run, the recipe, recipe properties, and ruby CIPD package did not change around that time at all.

[godofredoc]

I took a quick look, removing the installation of ruby makes the gems steps to succeed but all the tests fail.

E.g. for x64 GEM_HOME is set to a generic folder where blundler and bundle are installed https://cs.opensource.google/flutter/recipes/+/main:recipe_modules/flutter_deps/api.py;l=300 and only after the gems path has been set and gems have been installed the path is changes to the ruby specific location https://cs.opensource.google/flutter/recipes/+/main:recipe_modules/flutter_deps/api.py;l=326

In arm64 GEM_HOME is set directly to <gem_dir>/ruby/3.1.0 when the bundler and bundle are located at <gem_dir>

Another potential problem is that ruby binaries from cipd are installed at the end of the path and most likely never used is ruby is already installed in the system or the code is running from an xcode context. https://cs.opensource.google/flutter/recipes/+/main:recipe_modules/flutter_deps/api.py;l=282

[keyonghan]

Compared the failed build with succeeded one, the env. dir path root was switched from /opt to /Volumes/Work, which caused issues. Followed up in the chat to see why such changes in our bots.

[keyonghan]

Chromium team reinstalled our bots on 10/07 morning, which is aligned with our incident here. It is suspected some hard coded paths somewhere.

[godofredoc]

Apart from the what actually caused this problem. We need to make this installation hermetic and resilience to changes in the environment.

[keyonghan]

Yeah, I agree. When creating the ruby version, a prefix path is needed to make it work. I believe that's the culprit here.

[keyonghan]

Have spent quite some hours to work on the portable version, but still no luck to make it work.
Referenced https://github.com/stephan-nordnes-eriksen/ruby_ship, which seems promising to meet our purpose. However this still depends on openssl pre-installed in the bot, which will make the ruby doesn't work again if the root dir changes.

I am running out of ideas.. Anyone with experience on ruby may have a better idea on how to make it work? /cc @jmagman

One possible workaround is build/deploy openssl and ruby on the runtime, but it will add 5+ mins on each run.

[jmagman]

Were you able to figure out why Install gems is working for other arm builders, like
https://ci.chromium.org/p/flutter/builders/prod/Mac_arm64_ios%20build_ios_framework_module_test/2149
https://ci.chromium.org/p/flutter/builders/prod/Mac_arm64_ios%20flutter_gallery_ios__compile/3432

[godofredoc]

Devicelab vs chromium machines?

[keyonghan]

Yeah, these are devicelab testbeds where the root dirs are not changed (still /opt/s), whereas the chromium ones have been updated to /Volumes/Work/s. The existing ruby was compiled based on /opt/s, so the devicelab ones are still valid.

[godofredoc]

@yusuf-goog FYI

[godofredoc]

This is fixed now with ruby hermetic packages and https://flutter-review.googlesource.com/c/recipes/+/38088

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.