RDMA issue: failed to create extended queue pair (QP): Operation not supported

[lewtun]

🐛 Describe the bug

Hello torchforgers!

I've successfully run the install command on 1 node of 8 x H100s, but when I then try to run the GRPO example with:

python -m apps.grpo.main --config apps/grpo/qwen3_1_7b.yaml

I get a peculiar RDMA error:

[RLTrainer-0/1] 2025-10-22 23:21:36 INFO Pushing weights for policy version 1
failed to create extended queue pair (QP): Operation not supported
[0]E1022 23:21:36.634458 2876119 hyperactor/src/proc.rs:1175] unix:@JEdzy6nlnlU4L0ytzv3i4UOA,anon_0_1BsCGXqAsYRV,rdma_manager[0]: actor failure: serving unix:@JEdzy6nlnlU4L0ytzv3i4UOA,anon_0_1BsCGXqAsYRV,rdma_manager[0]: processing error: could not create loopback QP for device rdmap79s0: failed to create queue pair (QP): Invalid argument (os error 22)

Any idea what might cause this?

Full stack trace attached:
forge-bug.txt

Versions

No response

[allenwang28]

Hey @lewtun! Thanks for opening this issue, could you please share what Cloud you're on? This error likely comes from the RDMA stack in Monarch which was mostly configured/tested to run on CoreWeave and Meta's internal stack so far. I'd like to forward this to the Monarch team.

In the meantime, a workaround could be to disable RDMA, which will save and load checkpoints via DCP locally, you can do this with

export TORCHSTORE_RDMA_ENABLED=0

[lewtun]

Thanks @allenwang28, the workaround works :)

We're running on a SageMaker Hyperpod cluster with P5 instances. We are using AWS EFA which relies on libfabric and not on libverbs which is the stack for Infiniband.

[casteryh]

Thanks @allenwang28, the workaround works :)

We're running on a SageMaker Hyperpod cluster with P5 instances. We are using AWS EFA which relies on libfabric and not on libverbs which is the stack for Infiniband.

Hi @lewtun , currently monarch RDMA only supports libibverbs. I encourage you to also file this as a feature request at https://github.com/meta-pytorch/monarch.

[casteryh]

Second thought: RDMA shouldn't have been enabled in the first place if no libibverbs devices are available on the machine.
@lewtun Can you run ibstat on your node and see what happens?

[lewtun]

Can you run ibstat on your node and see what happens?

Running this returns nothing