[Core] ray.wait not actually wait until ready when the task is longer than 12 days

[Michaelvll]

What happened + What you expected to happen

For a task longer than 12 days, ray.wait will return an empty list of ready object refs after 10**6 seconds when timeout is not specified, which is about 11.5 days.

This is inconsistent with what ray.get will do when timeout is not specified.

Versions / Dependencies

ray==2.9.3 (but I suppose it happens for all the ray versions)
python 3.10
OS Ubuntu 20.04

Reproduction script

ray/python/ray/_private/worker.py

Lines 2852 to 2853 in 1ccf925

	timeout = timeout if timeout is not None else 10**6
	timeout_milliseconds = int(timeout * 1000)

Issue Severity

None

[jjyao]

Hi @Michaelvll what's the cluster setup. Does the task run on the same node where ray.wait is called?

[Michaelvll]

Hi @Michaelvll what's the cluster setup. Does the task run on the same node where ray.wait is called?

Yes, the task is run on the same node as the driver, but I believe this happens for multi-node cases as well, due to the code quoted above. ray.get does not have the issue.

[hongchaodeng]

If it is always set to 10**6 seconds, we probably keep it as is and not break any compatibility.

It makes sense to have some default timeout like that so that api call would not hang forever. Nonetheless, we should change the docs to mention this.