Skip to content

[WIP] proto#62772

Draft
edoakes wants to merge 31 commits intoray-project:masterfrom
edoakes:eoakes/arrow-proto
Draft

[WIP] proto#62772
edoakes wants to merge 31 commits intoray-project:masterfrom
edoakes:eoakes/arrow-proto

Conversation

@edoakes
Copy link
Copy Markdown
Collaborator

@edoakes edoakes commented Apr 20, 2026

No description provided.

edoakes added 4 commits April 10, 2026 06:36
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
@edoakes edoakes added the go add ONLY when ready to merge, run all tests label Apr 20, 2026
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a Flight-based object store for Ray to handle Arrow tables out-of-band, utilizing Arrow Flight RPC for remote transfers and process_vm_readv for high-performance same-node transfers. The implementation includes a C++ core, Cython wrappers, and integration into Ray's task output and serialization paths. Feedback highlights several critical issues: the use of destructive read semantics in the Flight server which conflicts with Ray's immutable object model, a lack of platform guards for Linux-specific system calls, and a busy-wait loop in the worker output logic. Additionally, concerns were raised regarding memory efficiency due to eager serialization, security risks from binding to all network interfaces, and potential runtime errors in the Bazel repository rule.

Comment thread src/ray/flight_store/arrow_flight_store.cc Outdated
Comment thread src/ray/flight_store/arrow_flight_store.cc Outdated
Comment thread python/ray/_raylet.pyx
Comment thread src/ray/flight_store/arrow_flight_store.cc Outdated
Comment thread src/ray/flight_store/arrow_flight_store.cc Outdated
Comment thread bazel/pyarrow_repo.bzl Outdated
edoakes added 24 commits April 20, 2026 11:25
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
edoakes added 3 commits April 24, 2026 07:00
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
@edoakes edoakes force-pushed the eoakes/arrow-proto branch from f184f9f to 6888dd9 Compare April 29, 2026 20:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

go add ONLY when ready to merge, run all tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant