[Core] Command ray start --head reports missing module async_timeout

[zigaLuksic]

What happened + What you expected to happen

When running ray start --head I get the following message:

2023-11-20 09:23:24,124 ERROR services.py:1329 -- Failed to start the dashboard , return code 1
2023-11-20 09:23:24,124 ERROR services.py:1354 -- Error should be written to 'dashboard.log' or 'dashboard.err'. We are printing the last 20 lines for you. See 'https://docs.ray.io/en/master/ray-observability/ray-logging.html#logging-directory-structure' to find where the log file is.
2023-11-20 09:23:24,124 ERROR services.py:1398 -- 
The last 20 lines of /tmp/ray/session_2023-11-20_09-23-22_261599_52502/logs/dashboard.log (it contains the error message from the dashboard): 
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zluksic/Documents/Projects/clean/lib/python3.11/site-packages/ray/dashboard/utils.py", line 134, in get_all_modules
    raise e
  File "/home/zluksic/Documents/Projects/clean/lib/python3.11/site-packages/ray/dashboard/utils.py", line 121, in get_all_modules
    importlib.import_module(name)
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/zluksic/Documents/Projects/clean/lib/python3.11/site-packages/ray/dashboard/modules/test/test_agent.py", line 6, in <module>
    import ray.dashboard.modules.test.test_utils as test_utils
  File "/home/zluksic/Documents/Projects/clean/lib/python3.11/site-packages/ray/dashboard/modules/test/test_utils.py", line 3, in <module>
    import async_timeout
ModuleNotFoundError: No module named 'async_timeout'

This started occurring on our scheduled tests during the weekend (Friday was fine).

Versions / Dependencies

Clean Python 3.11 environment with only ray[default] installed.

Reproduction script

• create new Python 3.11 environment
• run pip install ray[default]
• run ray start --head

Issue Severity

None

[anhnami]

Had the same issue, running 'pip install async-timeout' fixed it for me.

[garrison]

Ray depends on async-timeout, but it does not list it in its dependencies. Instead, it has relied it on being installed as a transitive dependency of aiohttp. However, aiohttp 3.9.0 was released a few days ago, which removed its dependency on async-timeout (aio-libs/aiohttp#7502) when under Python 3.11 and higher. So now Ray needs to depend on async-timeout explicitly if it is going to continue using it.

[jjyao]

Normal ray usage shouldn't import tests. We will fix it in 2.9 release.

[dss010101]

when is the 2.9 release targeted for?

[jjyao]

It's around mid December.

[rickyyx]

This is showing up in release test as well: #41372 (comment)

[ddelange]

is this something that qualifies for a bugfix/cherry-pick release on existing minor versions? Maybe even going back to 2.6?

[anyscalesam]

@ddelange for this issue unfortunately no; we reserve backporting to critical issues as we release forward fairly often. please upgrade to ray29 and/or mitigate via the workaround documented above.

[ddelange]

hi 👋 #41775 was not included in ray 2.9.0 which was just released. could someone cherry-pick it onto 2.9.1?