[client] Better version-compatibility validation

[richardliaw]

What happened + What you expected to happen

Currently users can often run into issues where version-compatibility issues for Ray Client can cause weird behaviors and hard to debug. If we can surface this issue more quickly and clearly, this can help make the experience better

Versions / Dependencies

Linux

Reproduction script

import ray

local_version = ray.__version__

ray.init("ray://localhost:10001")

@ray.remote
def get_version():
    return ray.__version__

remote_version = ray.get(get_version.remote())

matches = local_version == remote_version
if not matches:
    raise Exception(f"Ray version mismatch! local version is {local_version}, remote version is {remote_version}")
else:
    print(f"welcome to Ray Client! version {local_version}")

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[rynewang]

I took a look to the code.

• in a Ray worker it checks Python version and Ray version.
  • python/ray/_private/utils.py
  • check_version_info, called in connect()
• in a Ray client, it checks Python version and CURRENT_PROTOCOL_VERSION, which is set to "2023-06-27" rn.
  • python/ray/util/client/init.py
  • _ClientContext._check_versions

I am not sure where this discrepancy comes from.

[rynewang]

Based on the discussions, we would do this for Client:

• error on 2 digit python mismatch
• warning on 3 digit python mismatch
• error on ray version mismatch