[Bug] test_detached_actor does not handle messages from sockets properly

[kevin85421]

Search before asking

• I searched the issues and found no similar issues.

KubeRay Component

ci

What happened + What you expected to happen

kuberay/tests/compatibility-test.py

Lines 329 to 360 in 159d57f

	s._sock.sendall(b'''
	def get_detached_actor():
	return ray.get_actor("a")
	a = retry_with_timeout(get_detached_actor)
	def get_new_value():
	return ray.get(a.ready.remote())
	res2 = retry_with_timeout(get_new_value)
	if res1 != res2:
	print('successful: {} {}'.format(res1, res2))
	sys.exit(0)
	else:
	print('failed: {} {}'.format(res1, res2))
	raise Exception('failed')
	''')
	count = 0
	while count < 90:
	try:
	buf = s._sock.recv(4096)
	logger.info(buf.decode())
	if buf.decode().find('successful') != -1:
	break
	if buf.decode().find('failed') != -1:
	raise Exception('test failed {}'.format(buf.decode()))
	except Exception as e:
	pass
	time.sleep(1)
	count += 1
	if count >= 90:
	raise Exception('failed to run script')

L351 checks whether the received message includes "successful" or not. If the message includes "successful", it will leave the loop. Ideally, we expected the "successful" string is produced by L339. However, the received message is "Unable to connect to GCS at 10.244.0.8:6379. Check that (1) Ray GCS with matching version started successfully at the specified address, and (2) there is no firewall setting preventing access"

The following log is printed by L350.

INFO:__main__:Unable to connect to GCS at 10.244.0.8:6379. Check that (1) Ray GCS with matching version started successfully at the specified address, and (2) there is no firewall setting preventing access.

Reproduction script

python3 tests/compatibility-test.py RayFTTestCase.test_detached_actor 2>&1 | tee log

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[kevin85421]

cc @DmitriGekhtman

[DmitriGekhtman]

cc @wilsonwang371

[wilsonwang371]

good catch! I can make some change or are you planing to make a patch? @kevin85421

[kevin85421]

@wilsonwang371 I plan to make a patch. I hope to learn more about the KubeRay E2E tests by fixing this issue. Thank you!