[Dashboard] Ray 1.13.1 cannot start dashboard with grpc version 1.29.0

[jianoaix]

The 1.13.1 release set the required grpc version to be "grpcio >= 1.28.1, <= 1.43.0". However, A user who's running on 1.13.1 with grpc version 1.29.0 got the following error when they try to start Ray cluster:

2022-08-19 05:51:03,709	ERROR services.py:1488 -- Failed to start the dashboard: Failed to start the dashboard, return code 1
 The last 10 lines of /mnt/mesos/sandbox/ray_log/session_2022-08-19_05-51-01_217114_207/logs/dashboard.log:
  File "/usr/lib/python3.6/site-packages/ray/dashboard/modules/state/state_head.py", line 11, in <module>
    from ray.dashboard.state_aggregator import StateAPIManager
  File "/usr/lib/python3.6/site-packages/ray/dashboard/state_aggregator.py", line 21, in <module>
    from ray.experimental.state.state_manager import StateDataSourceClient
  File "/usr/lib/python3.6/site-packages/ray/experimental/state/state_manager.py", line 67, in <module>
    class StateDataSourceClient:
  File "/usr/lib/python3.6/site-packages/ray/experimental/state/state_manager.py", line 80, in StateDataSourceClient
    def __init__(self, gcs_channel: grpc.aio.Channel):
AttributeError: module 'grpc' has no attribute 'aio'

....

[2m[33m(raylet)[0m /usr/lib/python3.6/site-packages/ray/dashboard/modules/reporter/reporter_agent.py:45: UserWarning: `gpustat` package is not installed. GPU monitoring is not available. To have full functionality of the dashboard please install `pip install ray[default]`.)
[2m[33m(raylet)[0m   "`gpustat` package is not installed. GPU monitoring is "
[2m[33m(raylet)[0m [2022-08-19 05:51:05,137 E 285 333] (raylet) agent_manager.cc:107: The raylet exited immediately because the Ray agent failed. The raylet fate shares with the agent. This can happen because the Ray agent was unexpectedly killed or failed. See `dashboard_agent.log` for the root cause.
[2m[36m(pid=gcs_server)[0m [2022-08-19 05:51:05,306 E 212 212] (gcs_server) gcs_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: 
[2m[36m(pid=gcs_server)[0m [2022-08-19 05:51:06,306 E 212 212] (gcs_server) gcs_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: 
[2m[36m(pid=gcs_server)[0m [2022-08-19 05:51:07,306 E 212 212] (gcs_server) gcs_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: 
[2m[36m(pid=gcs_server)[0m [2022-08-19 05:51:08,306 E 212 212] (gcs_server) gcs_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: 
[2m[36m(pid=gcs_server)[0m [2022-08-19 05:51:09,306 E 212 212] (gcs_server) gcs_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: 

It appears the Dashboard is requiring a different grpc version than the rest of Ray.

cc @scv119

[scv119]

@alanwguo seems an issue with dashboard. are you the right owner of the problem? we could either fix the dashboard, or change the grpcio requirement.

[alanwguo]

@jianoaix , what version of python are you using? This might be related to #27269

[jianoaix]

@alanwguo python 3.6.

[rkooo567]

Hmm maybe we should bump the minimal version of grpcio. Is it possible to do this @jianoaix for the user?

[alanwguo]

@rkooo567 , i have a PR to address this: #28025