[Core] Support publishing Submission job events using ray event recorder#61099
Open
sampan-s-nayak wants to merge 36 commits into
Open
[Core] Support publishing Submission job events using ray event recorder#61099sampan-s-nayak wants to merge 36 commits into
sampan-s-nayak wants to merge 36 commits into
Conversation
Signed-off-by: sampan <sampan@anyscale.com>
Signed-off-by: sampan <sampan@anyscale.com>
Signed-off-by: sampan <sampan@anyscale.com>
Signed-off-by: sampan <sampan@anyscale.com>
Signed-off-by: sampan <sampan@anyscale.com>
Signed-off-by: sampan <sampan@anyscale.com>
…nto job_events_missing_fields_3 Signed-off-by: sampan <sampan@anyscale.com>
Signed-off-by: sampan <sampan@anyscale.com>
sampan-s-nayak
requested review from
a team,
edoakes and
jjyao
as code owners
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request successfully integrates the Ray event recorder framework into the job submission system, enabling the emission of job definition and lifecycle events. The implementation follows the existing pattern for actor and node events, using a priority-based approach where the new framework takes precedence when enabled. The addition of an end-to-end test ensures the correctness of event emission and capture. I have provided a few suggestions to improve the efficiency of protobuf timestamp handling and to ensure robust initialization of the event recorder.
Comment on lines
+144
to
+147
| now = time.time() | ||
| transition.timestamp.CopyFrom( | ||
| Timestamp(seconds=int(now), nanos=int((now % 1) * 1e9)) | ||
| ) |
Contributor
There was a problem hiding this comment.
Instead of creating a new Timestamp object and using CopyFrom, you can use the FromMilliseconds method directly on the existing transition.timestamp object. This is more efficient and avoids manual nanosecond calculations which can sometimes suffer from floating-point precision issues.
Suggested change
| now = time.time() | |
| transition.timestamp.CopyFrom( | |
| Timestamp(seconds=int(now), nanos=int((now % 1) * 1e9)) | |
| ) | |
| transition.timestamp.FromMilliseconds(int(time.time() * 1000)) |
Comment on lines
+207
to
+223
| if ray_constants.RAY_ENABLE_RAY_EVENT: | ||
| try: | ||
| from ray._raylet import initialize_event_recorder | ||
|
|
||
| initialize_event_recorder( | ||
| aggregator_address=self._dashboard_agent.ip, | ||
| aggregator_port=self._dashboard_agent.grpc_port, | ||
| node_ip=self._dashboard_agent.ip, | ||
| node_id_hex=self._dashboard_agent.node_id, | ||
| max_buffer_size=10000, | ||
| ) | ||
| logger.info("Initialized ray event recorder in JobAgent.") | ||
| except Exception: | ||
| logger.warning( | ||
| "Failed to initialize ray event recorder in JobAgent.", | ||
| exc_info=True, | ||
| ) |
Contributor
There was a problem hiding this comment.
The initialize_event_recorder call is placed within the run method of JobAgent. While this works for initialization, consider if a corresponding shutdown_event_recorder call is needed when the agent or module stops, similar to the implementation in JobSupervisor. This ensures that any buffered events are flushed before the process exits.
sampan-s-nayak
requested review from
MengjinYan and
israbbani
as code owners
Signed-off-by: sampan <sampan@anyscale.com>
…elds_3 Signed-off-by: sampan <sampan@anyscale.com>
…ss attribute Cython cdef class types are immutable — setting EventRecorder._instance at module scope raises TypeError at runtime. Use a module-level _event_recorder_instance variable with global declarations instead. Signed-off-by: sampan <sampan@anyscale.com>
…elds_3 Signed-off-by: sampan <sampan@anyscale.com>
Signed-off-by: sampan <sampan@anyscale.com>
Signed-off-by: sampan <sampan@anyscale.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
sampan-s-nayak
added
go
Signed-off-by: sampan <sampan@anyscale.com>
|
This pull request has been automatically marked as stale because it has not had You can always ask for help on our discussion forum or Ray's public slack channel. If you'd like to keep this open, just leave any comment, and the stale label will be removed. |
github-actions
Bot
added
the
stale
Contributor
Author
|
not stale |
Existing users who set RAY_enable_ray_event for C++ events should not automatically opt into the new Python event pipeline. Introduce a separate RAY_enable_python_ray_event flag for Python-side ONE-event publishing (job events via EventRecorder, autoscaler events via DashboardHeadRayEventPublisher). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: sampan <sampan@anyscale.com>
Signed-off-by: sampan <sampan@anyscale.com>
github-actions
Bot
added
unstale
Signed-off-by: Sampan S Nayak <sampansnayak2@gmail.com>
MengjinYan
added a commit
that referenced
this pull request
May 6, 2026
## Description Adds cython bindings for c++ rayEventRecorder and sets up the scaffolding required to emit events from python (using the new one-event framework). This will also be used to emit library events in the future refer: #61099 for example usage of this abstraction --------- Signed-off-by: sampan <sampan@anyscale.com> Signed-off-by: Sampan S Nayak <sampansnayak2@gmail.com> Co-authored-by: sampan <sampan@anyscale.com> Co-authored-by: Mengjin Yan <mengjinyan3@gmail.com>
chillCode404
pushed a commit
to chillCode404/ray-contrib
that referenced
this pull request
May 9, 2026
## Description Adds cython bindings for c++ rayEventRecorder and sets up the scaffolding required to emit events from python (using the new one-event framework). This will also be used to emit library events in the future refer: ray-project#61099 for example usage of this abstraction --------- Signed-off-by: sampan <sampan@anyscale.com> Signed-off-by: Sampan S Nayak <sampansnayak2@gmail.com> Co-authored-by: sampan <sampan@anyscale.com> Co-authored-by: Mengjin Yan <mengjinyan3@gmail.com>
am-kinetica
pushed a commit
to kineticadb/ray
that referenced
this pull request
May 14, 2026
## Description Adds cython bindings for c++ rayEventRecorder and sets up the scaffolding required to emit events from python (using the new one-event framework). This will also be used to emit library events in the future refer: ray-project#61099 for example usage of this abstraction --------- Signed-off-by: sampan <sampan@anyscale.com> Signed-off-by: Sampan S Nayak <sampansnayak2@gmail.com> Co-authored-by: sampan <sampan@anyscale.com> Co-authored-by: Mengjin Yan <mengjinyan3@gmail.com> Signed-off-by: anindyam1969 <amukherjee@kinetica.com>
Lucas61000
pushed a commit
to Lucas61000/ray
that referenced
this pull request
May 15, 2026
## Description Adds cython bindings for c++ rayEventRecorder and sets up the scaffolding required to emit events from python (using the new one-event framework). This will also be used to emit library events in the future refer: ray-project#61099 for example usage of this abstraction --------- Signed-off-by: sampan <sampan@anyscale.com> Signed-off-by: Sampan S Nayak <sampansnayak2@gmail.com> Co-authored-by: sampan <sampan@anyscale.com> Co-authored-by: Mengjin Yan <mengjinyan3@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
use the newly added python ray event exporter framework to emit submission job events and verify using an E2E test.