Skip to content

Commit

Permalink
Documentation: Add usage example for verbs import
Browse files Browse the repository at this point in the history
Add a documentation for shared PD and verbs import usage in Pyverbs.
This includes a code snippet to demonstrate a usage example.

Reviewed-by: Ido Kalir <idok@mellanox.com>
Signed-off-by: Edward Srouji <edwards@mellanox.com>
  • Loading branch information
EdwardSro authored and yishaih committed Jul 9, 2020
1 parent 50d470c commit 3790121
Showing 1 changed file with 40 additions and 0 deletions.
40 changes: 40 additions & 0 deletions Documentation/pyverbs.md
Expand Up @@ -626,3 +626,43 @@ ctx = Context(name='rocep0s8f0')
uar = Mlx5UAR(ctx)
uar.close()
```

##### Import device, PD and MR
Importing a device, PD and MR enables processes to share their context and then
share PDs and MRs that is associated with.
A process creates a device and then uses some of the Linux systems calls to dup
its 'cmd_fd' member which lets other process to obtain ownership.
Once other process obtains the 'cmd_fd' it can import the device, then PD(s) and
MR(s) to share these objects.
Like in C, Pyverbs users are responsible for unimporting the imported objects
(which will also close the Pyverbs instance in our case) after they finish using
them, and they have to sync between the different processes in order to
coordinate the closure of the objects.
Unlike in C, closing the underlying objects is currently supported only via the
"original" object (meaning only by the process that creates them) and not via
the imported object. This limitation is made because currently there's no
reference or relation between different Pyverbs objects in different processes.
But it's doable and might be added in the future.
Here is a demonstration of importing a device, PD and MR in one process.
```python
from pyverbs.device import Context
from pyverbs.pd import PD
from pyverbs.mr import MR
import pyverbs.enums as e
import os

ctx = Context(name='ibp0s8f0')
pd = PD(ctx)
mr = MR(pd, 100, e.IBV_ACCESS_LOCAL_WRITE)
cmd_fd_dup = os.dup(ctx.cmd_fd)
improted_ctx = Context(cmd_fd=cmd_fd_dup)
imported_pd = PD(improted_ctx, handle=pd.handle)
imported_mr = MR(imported_pd, handle=mr.handle)
# MRs can be created as usual on the imported PD
secondary_mr = MR(imported_pd, 100, e.IBV_ACCESS_REMOTE_READ)
# Must manually unimport the imported objects (which close the object and frees
# other resources that use them) before closing the "original" objects.
# This prevents unexpected behaviours caused by the GC.
imported_mr.unimport()
imported_pd.unimport()
```

0 comments on commit 3790121

Please sign in to comment.