Skip to content

Conversation

@Andy-Jost
Copy link
Contributor

Enable IPC Buffer Re-export and Add Buffer.is_mapped Property

Summary

This PR enhances the CUDA IPC (Inter-Process Communication) functionality to support re-exporting buffers that were originally imported from another process. It also introduces a new Buffer.is_mapped property to track whether a buffer was mapped into the current process via IPC.

Key Changes

1. New Buffer.is_mapped Property

  • Added is_mapped property to Buffer class to indicate if a buffer was imported via IPC
  • Returns True for buffers imported via Buffer.from_ipc_descriptor()
  • Returns False for locally-allocated buffers

2. Enable Re-export of Imported Buffers

  • Removed restriction: Previously, attempting to export a memory resource that was imported via IPC would raise RuntimeError("Imported memory resource cannot be exported")
  • Buffers can now be re-exported after being imported, enabling multi-hop IPC chains (e.g., Process A → Process B → Process C)
  • IPC descriptors are now cached on first export to avoid redundant operations

3. IPC Data Structure Improvements

  • Rename IPCData into IPCDataForMR: Tracks IPC state for memory resources (allocation handle + mapped flag)
  • Added IPCDataForBuffer: Tracks IPC state for individual buffers (descriptor + mapped flag)
  • Added _ipc_data member to Buffer class to track IPC-related metadata
  • IPC descriptors are cached on buffers to support efficient re-export

4. Enhanced Testing

  • New test: TestIpcReexport validates the 3-process chain scenario:
    1. Process A allocates a buffer and exports it to Process B
    2. Process B receives the buffer and re-exports it to Process C
    3. Process C receives the buffer, modifies it, and signals completion
    4. Process A verifies that C's modifications are visible
  • Added is_mapped assertions throughout IPC tests
  • Converted test_ipc_send_buffers to class-based test structure
  • Removed obsolete TestExportImportedMR error test (no longer an error)

@Andy-Jost Andy-Jost added this to the cuda.core beta 10 milestone Dec 2, 2025
@Andy-Jost Andy-Jost requested review from leofang and rparolin December 2, 2025 20:10
@Andy-Jost Andy-Jost self-assigned this Dec 2, 2025
@Andy-Jost Andy-Jost added P0 High priority - Must do! feature New feature or request cuda.core Everything related to the cuda.core module labels Dec 2, 2025
@copy-pr-bot
Copy link
Contributor

copy-pr-bot bot commented Dec 2, 2025

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@Andy-Jost
Copy link
Contributor Author

/ok to test c82c120

@github-actions

This comment has been minimized.

@Andy-Jost Andy-Jost requested a review from rwgk December 2, 2025 22:33
@Andy-Jost
Copy link
Contributor Author

/ok to test b363790

Copy link
Collaborator

@rwgk rwgk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

self._ptr_obj = ptr
self._size = size
self._memory_resource = mr
self._ipc_data = IPCDataForBuffer(ipc_descriptor, True) if ipc_descriptor is not None else None
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: This would be more readable with is_mapped=True.

I'd do that systematically for all IPCDataForBuffer and IPCDataForMR calls (I think there are only four calls).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I actually had exactly that initially but noticed that Cython passes the keyword arguments as true keywords (in a dict) -- I had hoped that it would perform some magic, since the call target is a cdef function. Within this project, it's hard to know how to trade readability off against performance.

@Andy-Jost Andy-Jost merged commit 1d8b02e into NVIDIA:main Dec 2, 2025
61 checks passed
@Andy-Jost Andy-Jost deleted the multi-gpu-with-ipc branch December 2, 2025 23:50
@github-actions
Copy link

github-actions bot commented Dec 3, 2025

Doc Preview CI
Preview removed because the pull request was closed or merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cuda.core Everything related to the cuda.core module feature New feature or request P0 High priority - Must do!

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants