Skip to content

[DirectVerbs] Add mlx5_dv transport with create/modify/destroy QP support #104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
pakmarkthub wants to merge 45 commits into
base: devel-dv
Choose a base branch
from
Open

[DirectVerbs] Add mlx5_dv transport with create/modify/destroy QP support #104

pakmarkthub wants to merge 45 commits into from

Conversation

@pakmarkthub
Copy link
Contributor

@pakmarkthub pakmarkthub commented Oct 19, 2021

Prerequisite:

This PR:

  • introduces mlx5-dv transport module.
  • implements create/modify/destroy QP in the mlx5-dv transport.
  • implements create/destroy CQ in the mlx5-dv transport.
  • adds support for RC and UC connections.
  • introduces gds_modify_qp. We use DEVX to create QPs and they are not compatible with ibv_modify_qp.

Presubmit testing:

  • on gc11. Compiled successfully. Could run pass gds_create_qp but segfault'd on other parts because we haven't implemented them yet.

@pakmarkthub
Added --with-spectrum-mpi to configure
ce78eb6
@pakmarkthub
Introduced GDS_DRIVER_TYPE to gds_qp and gds_cq
66409ac
@pakmarkthub
Set gds_qp and gds_cq dtype to MLX5_EXP in the creation functions
fdd2270
@pakmarkthub
Changed the definition of gds_qp_init_attr_t and gds_send_wr. Also, r…
0073f81 
…emoved res_domain from the gds_qp struct
@pakmarkthub
Implemented gds_get_driver_type
78eadbb
@pakmarkthub
Initial implementation of mlx5-exp.cpp/hpp and moved create/destroy q…
aade2ab 
…p/cq functions to mlx5-exp
@pakmarkthub
Implemented gds_mlx5_exp_destroy_cq
5e789db
@pakmarkthub
Implemented gds_destroy_qp and gds_destroy_cq by connecting to gds_ml…
2a8ac7d 
…x5_exp_*
@pakmarkthub
Reimplemented gds_create_qp by connecting to gds_mlx5_exp_create_qp
a9351f2
@pakmarkthub
Moved gds_send_wr from exp-verbs to ib-verbs in gds_kernel_latency.c
e188668
@pakmarkthub
Fixed compile issues in gdsync.cpp and mlx5-exp.hpp
07d03a0
@pakmarkthub
Added mlx5-exp.cpp to the compile list
173841f
@pakmarkthub
Moved gds_prepare_send to gds_mlx5_exp_* and fixed compile errors in …
8d1ed50 
…mlx5-exp.*
@pakmarkthub
Modified gds_kernel_* applications to fit the new API/structs
10e164e
@pakmarkthub
Moved gds_wait_request to gds_mlx5_exp_wait_request and made the form…
b1cab05 
…er an opaque struct
@pakmarkthub
Moved gds_wait_request related functions to mlx5-exp
17b246f
@pakmarkthub
Changed the definition of gds_send_request_t
6be27ac
@pakmarkthub
Added gds_mlx5_exp_send_request_t definition and supported functions
f026578
@pakmarkthub
Moved APIs that use gds_send_request_t to mlx5-exp.cpp
92b09f0
@pakmarkthub
Removed include verbs_exp.h and peer_ops.h from public header files
e088b01
@pakmarkthub
Replaced structs related to ibv_exp_ with gds_* structs
942aa89
@pakmarkthub
Removed include verbs_exp.h and peer_ops.h from objs.hpp
35db866
@pakmarkthub
Removed include verbs_exp.h from objs.cpp
24a0a20
@pakmarkthub
Removed include verbs_exp.h and peer_ops.h from gdsync.cpp
5d7c328
@pakmarkthub
Relaced all IBV_EXP_ enum with GDS_ in gdsync.cpp
bf1a358
@pakmarkthub
Replaced all IBV_EXP_ enum with GDS_ in mlx5.cpp
f95c72d
@pakmarkthub
Relaced all IBV_EXP_ enum with GDS_ enum in gdsync_debug_hostregister…
6a23931 
…_bug.cpp
@pakmarkthub
Fixed bugs in gds_dump_wait_request
ab745de
@pakmarkthub
Updated some functions in gdsync_debug_hostregister_bug.cpp to reflec…
f43f905 
…t changes in gdsync.cpp
@pakmarkthub
Removed include verbs_exp.h and peer_ops.h from all files except from…
aeb1ed7 
… mlx-exp.*
@pakmarkthub
Updated configure.ac and Makefile.am to make exp-verbs optional
59d20f5
@pakmarkthub
Fixed bug in configure.ac related to checking for exp-verbs
bfb683b
@pakmarkthub
Defined gds_transport_t
e61a405
@pakmarkthub
Moved mlx5-exp.* into transports/mlx5-exp and implemented gds_transpo…
8720c86 
…rt_mlx5_exp_init
@pakmarkthub
Fixed bugs and modified the code to use the gds_transport_t interface
de7d52d
@pakmarkthub
Removed driver_type and fixed bugs
c914e89
@pakmarkthub
Added exp-verbs support checking in transport.hpp
ed703cc
@pakmarkthub
Fixed typo in configure.ac
7340483
@pakmarkthub
Fixed compile errors when compiling on x86
9986794
@pakmarkthub
Added initial implementation of DEVX and DirectVerbs QP/CQ create/des…
1ea8492 
…troy
@pakmarkthub
Connected dv destroy_qp to the transport interface
1c302b4
@pakmarkthub
Introduced gds_modify_qp
97a5f81
@pakmarkthub
Fixed typo in the help text in gds_kernel_latency
c053bb3
@pakmarkthub
Implemented CQ buffer allocation on GPU memory using DirectVerbs
d4f1f41
@pakmarkthub
Added UD support to QP creation with DEVX
f0c5167
@pakmarkthub pakmarkthub requested a review from drossetti October 19, 2021 07:14
@pakmarkthub pakmarkthub mentioned this pull request Oct 22, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@drossetti drossetti Awaiting requested review from drossetti

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@pakmarkthub