[Bug] Invalid NICs detected in `_get_rdma_devices()`

[alogfans]

https://github.com/MoonshotAI/checkpoint-engine/blob/05827aadf5a5e0fa6203b7ad41eef393a5f4d07a/checkpoint_engine/ps.py#L272C1-L312C19

Currently, the code looks up a specific directory to obtain NIC devices. However in my environment, an disabled (I'm not sure as I'm not the system maintainer, but the device does not exist in the output of ibv_device). I advise using ibv_get_device_list (or equivalent API methods) to retrieve the list of valid RDMA NIC devices. This ensures better portability, correctness, and alignment with the official API.

[weixiao-huang]

Thanks for your suggestion, is the code below match your expectation? Or is there a more simple code to get device list?

def _ibv_get_device_list() -> list[str]:
    import ctypes

    lib = ctypes.CDLL("libibverbs.so.1")
    lib.ibv_get_device_list.argtypes = [ctypes.POINTER(ctypes.c_int)]        # int *num_devices
    lib.ibv_get_device_list.restype  = ctypes.POINTER(ctypes.c_void_p)       # struct ibv_device **

    lib.ibv_free_device_list.argtypes = [ctypes.POINTER(ctypes.c_void_p)]
    lib.ibv_get_device_name.argtypes = [ctypes.c_void_p]              # struct ibv_device *
    lib.ibv_get_device_name.restype  = ctypes.c_char_p                # const char *


    num = ctypes.c_int()
    dev_array = lib.ibv_get_device_list(ctypes.byref(num))
    if not dev_array or num.value <= 0:
        return []

    devices = []
    for i in range(num.value):
        dev_ptr = dev_array[i]              # struct ibv_device *
        name    = lib.ibv_get_device_name(dev_ptr)  # const char *
        devices.append(name.decode())
    lib.ibv_free_device_list(dev_array)
    return devices

[weixiao-huang]

A more elegant way is to get the topology structures for GPU and RDMA net devices. We should find the nearest RDMA device for the GPU device in current rank

[alogfans]

Is it possible to just reading the output of ibv_devices?

[weixiao-huang]

Is it possible to just reading the output of ibv_devices?

ibv_devices may be not found in user's environment but libibverbs.so.1 should exist while using mooncake-transfer-engine

[alogfans]

@weixiao-huang OK, API hooking seems better.

[weixiao-huang]

@alogfans plz check whether this PR can get correct RDMA devices in your situation