Release test compiled_graphs failed

[can-anyscale]

Release test compiled_graphs failed. See https://buildkite.com/ray-project/release/builds/45135#01975883-d638-4329-9a82-3ffac7846b1d for more details.

Managed by OSS Test Policy

[can-anyscale]

Blamed commit: 63ea48b found by bisect job https://buildkite.com/ray-project/release-tests-bisect/builds/2494

[kevin85421]

Anyscale fails to start, and the error message is also related to AWS S3. It doesn’t seem to be related to my PR.

[can-anyscale]

Test has been failing for far too long. Jailing.

[kevin85421]

https://buildkite.com/ray-project/release/builds/45321/summary/annotations?sid=01975da4-0876-4b6e-9c60-028617f23e1e#01975dac-bae1-4994-9abc-41b1cf6372f0/532-582

I think the issue is from #53395. cc @ruisearch42 would you mind taking a look at this issue? Thanks!

[hipudding]

Could you give me the testcase name? I didn't have access to this release build logs. @kevin85421

[kevin85421]

+ python run_microbenchmark.py --experimental
--
  | Tip: set TESTS_TO_RUN='pattern' to run a subset of benchmarks
  | 2025-06-10 23:30:43,944	INFO worker.py:1595 -- Using address local set in the environment variable RAY_ADDRESS
  | 2025-06-10 23:30:46,428	INFO worker.py:1907 -- Started a local Ray instance. View the dashboard at https://session-xr25xdhkjc78ivlj97y7smdz4d.i.anyscaleuserdata-staging.com
  | [unstable] local put:local get, single channel calls per second 78408.71 +- 503.39
  | [unstable] local put:1 remote get, single channel calls per second 90075.81 +- 1188.84
  | Testing multiple readers/channels, n=32
  | [unstable] local put:n remote get, single channel calls per second 2670.9 +- 14.88
  | [unstable] local put:1 remote get, n channels calls per second 4356.35 +- 49.3
  | [unstable] local put:n remote get, n channels calls per second 4629.62 +- 414.1
  | [unstable] single-actor DAG calls per second 985.52 +- 15.19
  | Traceback (most recent call last):
  | File "/tmp/ray/session_2025-06-10_23-30-28_751234_2950/runtime_resources/working_dir_files/s3_ray-release-automation-results_working_dirs_compiled_graphs_rgjghctrqu__anyscale_pkg_2bc896f73269fb15da56a7be0c29fc3d/run_microbenchmark.py", line 30, in <module>
  | results = main() or []
  | File "/home/ray/anaconda3/lib/python3.9/site-packages/ray/_private/ray_experimental_perf.py", line 179, in main
  | compiled_dag = dag.experimental_compile()
  | File "/home/ray/anaconda3/lib/python3.9/site-packages/ray/dag/dag_node.py", line 340, in experimental_compile
  | return build_compiled_dag_from_ray_dag(
  | File "/home/ray/anaconda3/lib/python3.9/site-packages/ray/dag/compiled_dag_node.py", line 3310, in build_compiled_dag_from_ray_dag
  | compiled_dag._get_or_compile()
  | File "/home/ray/anaconda3/lib/python3.9/site-packages/ray/dag/compiled_dag_node.py", line 1575, in _get_or_compile
  | self._preprocess()
  | File "/home/ray/anaconda3/lib/python3.9/site-packages/ray/dag/compiled_dag_node.py", line 1302, in _preprocess
  | self._init_communicators()
  | File "/home/ray/anaconda3/lib/python3.9/site-packages/ray/dag/compiled_dag_node.py", line 1328, in _init_communicators
  | accelerator_module_name = AcceleratorContext.get().module_name
  | File "/home/ray/anaconda3/lib/python3.9/site-packages/ray/experimental/channel/accelerator_context.py", line 76, in get
  | _default_accelerator_context = AcceleratorContext(
  | File "/home/ray/anaconda3/lib/python3.9/site-packages/ray/experimental/channel/accelerator_context.py", line 41, in __init__
  | self._torch_mod = importlib.import_module(f"torch.{torch_module_name}")
  | File "/home/ray/anaconda3/lib/python3.9/importlib/__init__.py", line 127, in import_module
  | return _bootstrap._gcd_import(name[level:], package, level)
  | File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  | File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  | File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  | File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  | File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  | File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  | File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
  | ModuleNotFoundError: No module named 'torch'
  | Subprocess return code: 1