Skip to content

feat(admin): recover unhealthy Ray client connections - #1256

Merged
zhangjaycee merged 1 commit into
alibaba:masterfrom
zhangjaycee:feat/ray-disconnect-healthcheck
Jul 20, 2026
Merged

feat(admin): recover unhealthy Ray client connections#1256
zhangjaycee merged 1 commit into
alibaba:masterfrom
zhangjaycee:feat/ray-disconnect-healthcheck

Conversation

@zhangjaycee

Copy link
Copy Markdown
Collaborator

close #1255

  • Probe both local Ray initialization and cluster_resources RPC health before applying rotation policies.
  • Reuse the serialized shutdown/init retry path when the current client session is closed, cleaned up, or unresponsive.
  • Run health checks every 60 seconds with a configurable 30-second RPC timeout and detailed success/failure logs.
  • Add unit coverage for unhealthy-session reconnects, cleaned-session errors, and healthy RPC responses.

- Probe both local Ray initialization and cluster_resources RPC health before applying rotation policies.
- Reuse the serialized shutdown/init retry path when the current client session is closed, cleaned up, or unresponsive.
- Run health checks every 60 seconds with a configurable 30-second RPC timeout and detailed success/failure logs.
- Add unit coverage for unhealthy-session reconnects, cleaned-session errors, and healthy RPC responses.
@Dengsheng-wzh

Copy link
Copy Markdown
Collaborator

LGTM

@xieyang0827
xieyang0827 self-requested a review July 20, 2026 11:16
@zhangjaycee
zhangjaycee merged commit 68988cd into alibaba:master Jul 20, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Detect and recover unhealthy Ray Client connections

3 participants