Skip to content

[BUG] ibverbs raw RX fails CREATE_RQ when real-time CQ timestamps are unsupported #283

Description

@dleshchev

Describe the bug

The ibverbs raw-RX benchmark on main fails during daqiri_init() while creating the DevX receive queue. The same raw-RX configuration initializes successfully with an image built before #281.

The failure is an initialization regression, not a throughput result:

[CRITICAL] CREATE_RQ (DevX) failed: Remote I/O error (syndrome 0x43d98f)

#281 made the RQ request MLX5_RQC_TIMESTAMP_FORMAT_REAL_TIME unconditionally. The affected device does not advertise MLX5DV_CONTEXT_FLAGS_REAL_TIME_TS, and its firmware rejects that unsupported RQ timestamp format.

Individually restoring the other RX-side changes did not resolve the failure:

  • Restoring the previous striding-RQ geometry still fails with syndrome 0x43d98f.
  • Restoring the previous RX WQ UMEM access flags still fails with syndrome 0x43d98f.
  • Restoring both still fails.
  • Omitting only the unsupported real-time timestamp request allows the identical configuration to initialize.

Steps/Code to reproduce bug

  1. Build current main in the project container with:

    BASE_TARGET=dpdk DAQIRI_ENGINE="dpdk ibverbs" scripts/build-container.sh
  2. On the affected ARM64 system, run daqiri_bench_raw_gpudirect in RX-only mode with a valid single-port raw ibverbs configuration using a GPU memory region and one striding RX queue.

  3. Observe that daqiri_init() fails at DevX CREATE_RQ before traffic or throughput measurement begins.

  4. Run the same configuration with a pre-ibverbs improvements #281 image and observe successful initialization.

No machine-local interface names, PCI addresses, MAC addresses, network addresses, or hostnames are required to reproduce the issue; substitute locally valid values in the example raw-RX configuration.

Expected behavior

Raw ibverbs RX initialization should create the DevX RQ successfully on devices that do not support real-time CQ timestamps. DAQIRI should request the real-time format only when mlx5dv_query_device() advertises MLX5DV_CONTEXT_FLAGS_REAL_TIME_TS; otherwise it should retain the device-clock CQ format and convert timestamps through the existing clock-conversion path.

The valid #281 changes should remain intact: RX WQ UMEM registration with no remote-access permissions, the reduced 64-stride region geometry, and the adapter-relative log_wq_pg_sz encoding.

Environment overview (please complete the following information)

  • Environment location: bare-metal host; DAQIRI runs in the privileged project container
  • Method of DAQIRI install: project container built from source
  • Container build: BASE_TARGET=dpdk DAQIRI_ENGINE="dpdk ibverbs" scripts/build-container.sh

Environment details

  • OS: Ubuntu 24.04.4 LTS
  • Architecture/platform: ARM64 NVIDIA DGX Spark
  • Kernel: 6.17.0-1014-nvidia
  • Kernel page size currently measured on both validation systems: 4096 bytes
  • NIC: NVIDIA ConnectX-7, MT2910 family (15b3:1021)
  • NIC firmware: 28.45.4028
  • rdma-core, libibverbs, and providers: 2407mlnx52-1.2407061
  • DAQIRI: main containing dbf0c8b (regression); pre-ibverbs improvements #281 image succeeds

Additional context

The mlx5 WQ page-size field is encoded relative to the adapter's fixed 4-KiB page. It is distinct from the page size used to align/register the UMEM and was not the cause in the individual-revert matrix. Increasing the buffer count to restore the previous effective queue depth also does not fix initialization.

A narrow fix will capability-gate the RQ timestamp format and select the matching timestamp decode path per queue. Hardware reproduction is required because this failure occurs in firmware during DevX object creation.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions