[Core] Publish platform events via Ray Event Recorder by richabanker · Pull Request #63329 · ray-project/ray

richabanker · 2026-05-13T22:30:35Z

Description

Add support for publishing Platform events via the python ray event exporter framework

gemini-code-assist

Code Review

This pull request introduces the PlatformEventBuilder class to support infrastructure platform events, such as those from Kubernetes, and integrates it into the Ray dashboard's observability module. The changes include initializing the EventRecorder in the dashboard head and emitting events during processing callbacks. Feedback suggests allowing the EventRecorder to generate unique IDs for event updates to avoid deduplication issues, refactoring environment variable checks for efficiency, and moving imports out of the event processing hot path.

gemini-code-assist · 2026-05-13T22:32:12Z

+        if os.environ.get("RAY_ENABLE_PYTHON_RAY_EVENT", "False").lower() in (
+            "true",
+            "1",
+        ):
+            try:
+                from ray._common.observability.platform_events import (
+                    PlatformEventBuilder,
+                )
+                from ray._raylet import EventRecorder


Several modules are imported inside _process_event_callback for every event. Since this callback can be invoked frequently, these imports introduce unnecessary overhead. It is recommended to move these imports to the top of the file or at least outside the hot path of the callback.

cursor · 2026-05-13T22:36:06Z

+            timestamp_ns=int(time.time() * 1e9),
+        )
+        EventRecorder.emit(cython_event)
+        return True


E2E test never initializes EventRecorder on worker

High Severity

The remote task emit_test_platform_event calls EventRecorder.emit() on a worker process, but EventRecorder.initialize() is never called on that worker. EventRecorder is a per-process singleton, and the only initialize() call in the entire codebase is in platform_event_head.py's run() method, which runs in the dashboard head process — a completely different process. When the worker calls emit(), the singleton _event_recorder_instance is None, so emit_batch silently drops the event and returns False. The test will then timeout at the wait_for_condition check. The comment even says "explicitly initializes and emits" but the code only emits.

^{Reviewed by Cursor Bugbot for commit e7be190. Configure here.}

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{Reviewed by Cursor Bugbot for commit 1e59f09. Configure here.}

cursor · 2026-05-14T00:26:30Z

+                )
+                head_node_id_hex = (
+                    head_node_id_bytes.decode() if head_node_id_bytes else None
+                )


Duplicates existing get_head_node_id utility function

Low Severity

The logic to fetch the head node ID from GCS (lines 54–61) duplicates the existing get_head_node_id() utility in ray.dashboard.modules.job.utils, which performs the identical KV lookup with the same key, namespace, and timeout, and returns the decoded hex string or None.

^{Reviewed by Cursor Bugbot for commit 1e59f09. Configure here.}

richabanker · 2026-05-14T00:44:25Z

@sampan-s-nayak could you please help take a pass at this PR whenever possible for you? Thanks!

richabanker · 2026-05-14T00:45:21Z


    def _process_event_callback(self, ray_event: RayEvent):
-        """Callback running in the main asyncio loop to cache events."""
+        """Thread-safe entry point that dispatches event caching to the main asyncio loop."""


change driven by this comment

edoakes · 2026-05-15T15:40:09Z

@sampan-s-nayak PTAL

sampan-s-nayak · 2026-05-18T09:26:08Z

 RAY_EXPORT_EVENT_MAX_BACKUP_COUNT = env_bool("RAY_EXPORT_EVENT_MAX_BACKUP_COUNT", 20)

+# Enables emitting events through the Python EventRecorder (One-Event framework)
+# to the AggregatorAgent. When enabled, dashboard modules that support it emit


is this specific to dashboard modules? should we have a specific flag just for platform events? (I think we already had some config to enable platform events, can we reuse that?)

I actually tried to keep this from your PR here #61099 which I think added this env var to control whether the One-Event framework pipeline is used for Python event emission. The other platform specific env var is RAY_DASHBOARD_INGEST_PLATFORM_EVENTS but thats more for whether the k8s events should even be ingested/collected.

if we have a single flag for all python events then in the future we wont have a way to disable a specific event type. can we instead make the config a list of supported event types and enable Platform_events by default?

sampan-s-nayak · 2026-05-19T04:18:18Z

+            try:
+                from ray._raylet import EventRecorder
+
+                head_node_id_bytes = await self.gcs_client.async_internal_kv_get(


we can use get_head_node_id() here instead?

You mean from https://github.com/ray-project/ray/blob/master/python/ray/dashboard/modules/job/utils.py#L37? I could.. but that adds a cross-module dependency from the platform_events module onto the job module which is unrelated...

I could however add it in ray/dashboard/utils.py and make both platform_events and job modules use it, lmk if you prefer that

Signed-off-by: Richa Banker <richabanker@google.com>

richabanker requested review from a team, MengjinYan, dayshah and edoakes as code owners May 13, 2026 22:30

gemini-code-assist Bot reviewed May 13, 2026

View reviewed changes

cursor Bot reviewed May 13, 2026

View reviewed changes

richabanker force-pushed the platform-events-ray-event-recorder branch from 8718785 to b35d418 Compare May 13, 2026 22:40

cursor Bot reviewed May 13, 2026

View reviewed changes

Comment thread python/ray/dashboard/modules/aggregator/tests/test_ray_platform_events.py Outdated

richabanker force-pushed the platform-events-ray-event-recorder branch from b35d418 to bac7c35 Compare May 13, 2026 23:25

cursor Bot reviewed May 13, 2026

View reviewed changes

Comment thread python/ray/dashboard/modules/platform_events/platform_event_head.py

richabanker force-pushed the platform-events-ray-event-recorder branch from bac7c35 to 271db30 Compare May 13, 2026 23:36

cursor Bot reviewed May 13, 2026

View reviewed changes

Comment thread python/ray/dashboard/modules/platform_events/tests/test_platform_event_head.py Outdated

richabanker force-pushed the platform-events-ray-event-recorder branch 3 times, most recently from a8a923c to 1e59f09 Compare May 14, 2026 00:21

cursor Bot reviewed May 14, 2026

View reviewed changes

richabanker commented May 14, 2026

View reviewed changes

ray-gardener Bot added core Issues that should be addressed in Ray Core observability Issues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or Profiling community-contribution Contributed by the community labels May 14, 2026

edoakes assigned sampan-s-nayak May 15, 2026

sampan-s-nayak reviewed May 18, 2026

View reviewed changes

sampan-s-nayak reviewed May 19, 2026

View reviewed changes

Comment thread python/ray/dashboard/modules/platform_events/platform_event_head.py Outdated

richabanker force-pushed the platform-events-ray-event-recorder branch from 1e59f09 to aeeb1e1 Compare May 19, 2026 22:57

[Core] Publish platform events via Ray Event Recorder

07e69ed

Signed-off-by: Richa Banker <richabanker@google.com>

richabanker force-pushed the platform-events-ray-event-recorder branch from 64e7752 to 07e69ed Compare May 19, 2026 22:58

Conversation

richabanker commented May 13, 2026

Description

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist Bot May 13, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot May 13, 2026

Choose a reason for hiding this comment

E2E test never initializes EventRecorder on worker

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot May 14, 2026

Choose a reason for hiding this comment

Duplicates existing get_head_node_id utility function

Uh oh!

richabanker commented May 14, 2026

Uh oh!

richabanker May 14, 2026

Choose a reason for hiding this comment

Uh oh!

edoakes commented May 15, 2026

Uh oh!

sampan-s-nayak May 18, 2026

Choose a reason for hiding this comment

Uh oh!

richabanker May 19, 2026

Choose a reason for hiding this comment

Uh oh!

sampan-s-nayak May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sampan-s-nayak May 19, 2026

Choose a reason for hiding this comment

Uh oh!

richabanker May 19, 2026

Choose a reason for hiding this comment

Uh oh!

sampan-s-nayak May 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Duplicates existing `get_head_node_id` utility function

sampan-s-nayak May 20, 2026 •

edited

Loading