[Dashboard] Add platform events module with K8s event ingestion and caching

[richabanker]

Description

Add a new platform_events dashboard head module that introduces a modular, extensible Provider pattern for platform-specific event monitoring in Ray.

Key changes:

1. PlatformEventProvider: An abstract base class setting a strict type contract (Callable[[RayEvent], None]) for all platform-specific event sources
2. KubernetesEventProvider: Encapsulates all K8s client library and watch logic including
   • watching Kubernetes events related to Ray custom resources (RayCluster, RayJob, RayService)
   • converting them to RayEvent proto
3. PlatformEventsHead: Acts as a generic, platform-agnostic in-memory cache and REST controller

Coming next:

1. PlatformEventsHead will use the RayEventRecorder to publish events through the unified event framework
2. Expand KubernetesEventProvider to ingest generic K8s pod events (e.g., OOMKilled, Evicted)

Related issues

Additional information

Design doc

Tested on a GKE cluster (with some UI changes not included in this PR)

[richabanker]

cc @sampan-s-nayak @andrewsykim for initial feedback

[gemini-code-assist]

Code Review

This pull request introduces the PlatformEventsHead module to the Ray dashboard, which monitors Kubernetes events for Ray clusters, jobs, and services. The module watches for relevant K8s events in separate threads, converts them into RayEvent protobuf messages, and exposes them via a new REST API endpoint. Review feedback suggests improving the efficiency of event deduplication and eviction using OrderedDict, preventing potential race conditions when accessing resource versions across threads, and optimizing the conversion of protobuf messages to dictionaries for the API response.

[dancingactor]

Please fix linter to pass CI, thanks!

1. https://buildkite.com/ray-project/microcheck/builds/42305/steps/canvas?jid=019d5104-fecb-402f-be5d-a97966692d83
2. https://buildkite.com/ray-project/microcheck/builds/42305/steps/canvas?jid=019d5104-f6e3-4fb6-bfc8-623995dda395

[richabanker]

Thanks to some excellent points raised by @MengjinYan , I have refactored the PR to encapsulate all the K8s specific event-watching logic in k8s_provider.py while keeping platform_event_head.py lightweight for processing of the RayEvents (after receiving them from the k8s provider). This way, we can extend the setup to have any other platform_event provider have their own event-watching logic as a new provider that can be wired into the platform_event_head.py

Requesting another round of review for the changes.
FYI @andrewsykim @ryanaoleary

[MengjinYan]

Thanks for the refactoring! The change looks good! Only one minor comment that can be addressed in a followup PR.

[MengjinYan]

Minor: The docstring expects _process_event_callback to be called from the main asyncio loop. But the contract is not enforced from the head side. A potential fix can be to dispatch the callback directly to the main asyncio loop in the _process_event_callback itself.

It doesn't break the functionality for this PR so no need to address in this one. We can add it when integrating with the python event library

[edoakes]

Non-blocking comments, aside from using asyncio k8s client if possible to avoid spawning excessive threads. If this has already been considered, please note it in the PR description.

[edoakes]

is there no asyncio-compatible watch API?

[richabanker]

I dont think so.. recently some support for asyncio was added in kubernetes-client/python#2547, but the note in the PR description says "still missing dynamic client, watch, stream etc", so asyncio watch is not yet supported in the K8s python client. cc @yliaog for confirmation

[yliaog]

that's correct, not right now. that will be added in the next release.

[andrewsykim]

thanks @yliaog, let's follow-up to use the new asyncio client once there's an official release

[ryanaoleary]

did another pass and have no remaining comments besides what's already been mentioned - LGTM

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{Reviewed by Cursor Bugbot for commit ff47644. Configure here.}

[cursor]

Thread join timeout silently loses thread cleanup guarantee

Low Severity

The join_all_threads function uses a 1-second timeout per thread but doesn't check whether the join actually succeeded. If a thread is blocked on a long network call and doesn't terminate within the timeout, cleanup() completes while the thread is still alive and may attempt to call loop.call_soon_threadsafe after the event loop is torn down, potentially raising an unhandled RuntimeError in the background thread.

 

^{Reviewed by Cursor Bugbot for commit ff47644. Configure here.}