ibv_create_comp_channel fails with ENOTTY (Inappropriate ioctl for device)

[jalalmostafa]

Subject

I am trying to run clickhouse database server with libvma however the sockets are not being offloaded (fallback to OS) because ring_simple cannot create resources due to a failure in ibv_create_comp_channel (sets errno to 25 ENOTTY).
sockperf works perfectly but not clickhouse server.

The clickhouse server uses the following socket options: TCP_NODELAY, SO_REUSEADDR, SO_RCVTIMEO, SO_SNDTIMEO

Configuration:

• Product version 9.8.51
• OS Ubuntu 23.04 Kernel 6.2
• OFED 23.10-1.1.9.0
• Hardware: Mellanox Technologies MT28908 Family [ConnectX-6]

VMA Parameters

export VMA_TRACELEVEL=debug
export VMA_LOG_FILE=/home/user/libvma.log
export VMA_OFFLOADED_SOCKETS=1
export VMA_FORK=1
export VMA_LOG_COLORS=0
export VMA_LOG_DETAILS=0
export VMA_EXCEPTION_HANDLING=3
export VMA_TCP_NODELAY=1
export VMA_STATS_FD_NUM=1024
export VMA_BF=0 # disable blue flame
export VMA_RING_ALLOCATION_LOGIC_TX=10
export VMA_RING_ALLOCATION_LOGIC_RX=10

Logs

VMA DEBUG: ENTER: socket_internal(domain=AF_INET(2), type=(524289), protocol=0) = 120
      VMA DEBUG: srdr:789:socket_internal() ***** socket_internal g_p_fd_collection=0x7f85bc284700
      VMA DEBUG: fdc:210:addsocket() ***** addsocket Function Parameters: fd=120 domain=AF_INET(2) type=(524289) check_offload=1
      VMA DEBUG: fdc:211:addsocket() ***** addsocket sock_type=1 sock_flags=524288
      VMA DEBUG: fdc:214:addsocket() ***** addsocket create_offloaded_sockets() returns 1
      VMA DEBUG: fdc:227:addsocket() ***** addsocket is_valid_fd() returns 1
      VMA DEBUG: fdc:263:addsocket() TCP rules are either not consistent or instructing to use VMA.transport == USE_VMA
      VMA DEBUG: si_tcp[fd=120]:244:sockinfo_tcp() tcp socket created
      VMA DEBUG: si_tcp[fd=120]:253:sockinfo_tcp() new pcb 0x7f85bc2c89f8 pcb state 0
      VMA DEBUG: si[fd=120]:506:setsockopt() ret (1)
      VMA DEBUG: si_tcp[fd=120]:3591:setsockopt() IPPROTO_TCP - TCP_NODELAY
      VMA DEBUG: si_tcp[fd=120]:3601:setsockopt() (TCP_NODELAY) nagle: 1
      VMA DEBUG: si[fd=120]:1605:setsockopt_kernel() going to OS for setsockopt level 6 optname 1
      VMA DEBUG: si_tcp[fd=120]:309:sockinfo_tcp() TCP PCB FLAGS: 0x40
      VMA DEBUG: si[fd=120]:208:fcntl() going to OS for fcntl cmd=2, arg=0x1
      VMA DEBUG: ENTER: setsockopt(fd=120, level=1, optname=2)
      VMA DEBUG: si[fd=120]:474:setsockopt() SOL_SOCKET - 2
      VMA DEBUG: si[fd=120]:506:setsockopt() ret (1)
      VMA DEBUG: si_tcp[fd=120]:3664:setsockopt() SOL_SOCKET - SO_REUSEADDR
      VMA DEBUG: si_tcp[fd=120]:3674:setsockopt() (SO_REUSEADDR) val: 1
      VMA DEBUG: si[fd=120]:1605:setsockopt_kernel() going to OS for setsockopt level 1 optname 2
      VMA DEBUG: EXIT: setsockopt() returned with 0
      VMA DEBUG: ENTER: bind(fd=120, AF_INET, addr=192.168.20.2, port=9000)
      VMA DEBUG: ndtm[0x7f85bc295900]:307:get_net_device_val() Found ETH: Net Device: ens106np0 for addr: 192.168.20.2
      VMA DEBUG: si_tcp[fd=120]:2350:bind() socket bound
      VMA DEBUG: EXIT: bind() returned with 0
      VMA DEBUG: ENTER: listen(fd=120, backlog=4096)
      VMA DEBUG: match:428:get_family_by_instance_first_matching_rule() MATCHING program name: *, application-id: *
      VMA DEBUG: match:408:get_family_by_first_matching_rule() No matching rule. Using VMA (default)
      VMA DEBUG: match:468:__vma_match_tcp_server() MATCH TCP SERVER (LISTEN): => VMA
      VMA DEBUG: si_tcp[fd=120]:2392:prepareListen() TRANSPORT: VMA, sock state = 0
      VMA DEBUG: si[fd=120]:1393:attach_as_uc_receiver() Attaching using bind to ip rule
      VMA DEBUG: si[fd=120]:1397:attach_as_uc_receiver() Attached to specific local if: (192.168.20.2) addr: 192.168.20.2:9000
      VMA DEBUG: match:428:get_family_by_instance_first_matching_rule() MATCHING program name: *, application-id: *
      VMA DEBUG: match:408:get_family_by_first_matching_rule() No matching rule. Using VMA (default)
      VMA DEBUG: match:468:__vma_match_tcp_server() MATCH TCP SERVER (LISTEN): => VMA
      VMA DEBUG: si[fd=120]:630:attach_receiver() Attaching to dst:192.168.20.2:9000, src:0.0.0.0:0, proto:TCP, if:192.168.20.2
      VMA DEBUG: ndv[0x7f85bc240000]:982:reserve_ring() Creating new RING for allocation logic 10 profile 0 key 120 user address (nil) user length 0
      VMA DEBUG: ring[0x7f85b6d87d00]:56:print_val() 0: 0x7f85b6d87d00: parent (nil)
      VMA DEBUG: ndtm[0x7f85bc295900]:364:get_net_device_val() Found ETH: Net Device: ens106np0 for index: 11
      VMA DEBUG: STATS: 448:vma_stats_instance_create_ring_block() Added ring local=0x7f85b6d9fe98 shm=0x7f85bbda6240

      VMA DEBUG: ring_slave[0x7f85b6d87d00]:130:print_val() 11: 0x7f85b6d87d00: parent (nil) type RING_ETH
      VMA DEBUG: ndtm[0x7f85bc295900]:364:get_net_device_val() Found ETH: Net Device: ens106np0 for index: 11
      VMA DEBUG: ring_simple[0x7f85b6d87d00]:133:ring_simple() new ring_simple()
      VMA DEBUG: ring_simple[0x7f85b6d87d00]:134:ring_simple() **** m_parent->get_if_index() 11
      VMA DEBUG: ndtm[0x7f85bc295900]:364:get_net_device_val() Found ETH: Net Device: ens106np0 for index: 11
      VMA DEBUG: ndtm[0x7f85bc295900]:364:get_net_device_val() Found ETH: Net Device: ens106np0 for index: 11
      VMA DEBUG: ring_simple[0x7f85b6d87d00]:255:create_resources() 2: m_parent->get_if_index()=11 get_if_index()=11
      VMA DEBUG: ring_simple[0x7f85b6d87d00]:259:create_resources() ***** ibv_get_device_name(ibvctx->device)= mlx5_2
      VMA DEBUG: ring_simple[0x7f85b6d87d00]:263:create_resources() ibv_create_comp_channel for tx failed. m_p_tx_comp_event_channel = (nil) (errno=25 Inappropriate ioctl for device)
      VMA DEBUG: ring_simple[0x7f85b6d87d00]:166:~ring_simple() delete ring_simple()
VMA DEBUG: ring_simple[0x7f85b6d87d00]:213:~ring_simple() Tx buffer poll: free count = 0, sender_has = 0, total = 0, good accounting (0)
      VMA DEBUG: ring_simple[0x7f85b6d87d00]:217:~ring_simple() Tx WR num: free count = 0, total = 0, good accounting (0)
      VMA DEBUG: ring_simple[0x7f85b6d87d00]:220:~ring_simple() Rx buffer pool: 0 free global buffers available
      VMA DEBUG: ring_simple[0x7f85b6d87d00]:237:~ring_simple() queue of event completion elements is empty
      VMA DEBUG: ring_simple[0x7f85b6d87d00]:247:~ring_simple() delete ring_simple() completed
      VMA DEBUG: ring_slave[0x7f85b6d87d00]:130:print_val() 11: 0x7f85b6d87d00: parent (nil) type RING_ETH
      VMA DEBUG: STATS: 456:vma_stats_instance_remove_ring_block() Remove ring local=0x7f85b6d9fe98

      VMA DEBUG: ndv[0x7f85bc240000]:1383:create_ring() failed creating ring create event channel failed
      VMA DEBUG: si[fd=120]:762:create_nd_resources() Failed to reserve ring for allocation key allocation logic 10 profile 0 key 120 user address (nil) user length 0 on ip 192.168.20.2
      VMA DEBUG: si_tcp[fd=120]:2463:listen() Fallback the connection to os