[Ray Client] Unable to connect to Ray Cluster with ray.init

[peterhaddad3121]

What happened + What you expected to happen

Ray 1.12.1 was connecting through gRPC perfectly fine, after upgrading both the cluster and local SDK to 1.13.0 I am no longer able to connect.

The error I receive is:

ConnectionError: ray client connection timeout

It looks like non of the code really changed between versions in: ~/.local/lib/python3.9/site-packages/ray/util/client/worker.py:264

Any ideas of what might be occurring? The only change is upgrading between the two versions.

Versions / Dependencies

Remote Cluster 1.13.0
Local Python Ray SDK: 1.13.0
Python Version: 3.7.9

Reproduction script

import os
import ray
import numpy as np

os.environ["RAY_USE_TLS"] = "1"
os.environ["RAY_TLS_SERVER_CERT"] = "cert.crt"
os.environ["RAY_TLS_SERVER_KEY"] = "/key.key"
os.environ["RAY_TLS_CA_CERT"] = "ca.crt"

info = ray.init(
    address="ray://" + remote_address + ":433"
    runtime_env=runtime_env)

Issue Severity

High: It blocks me from completing my task.

[stale]

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

• If you'd like to keep the issue open, just leave any comment, and the stale label will be removed!
• If you'd like to get more attention to the issue, please tag one of Ray's contributors.

You can always ask for help on our discussion forum or Ray's public slack channel.

[stale]

Hi again! The issue will be closed because there has been no more activity in the 14 days since the last message.

Please feel free to reopen or open a new issue if you'd still like it to be addressed.

Again, you can always ask for help on our discussion forum or Ray's public slack channel.

Thanks again for opening the issue!

[Krout0n]

@peterhaddad3121 I asked about the same issue and try to change your import-order. It worked for me.
https://discuss.ray.io/t/how-do-i-connect-tls-enabled-head-node-in-python-script/8743