Cannot open openib btl component #5280
Labels
Comments
|
Can you run with |
|
Thank you for the quick reply. Is openmpi supposed to work if the only RDMA/infiniband software installed on the system is |
|
Is there a simple way to check if my openmpi installation includes |
|
yeah - just use "ompi_info" and you'll see the list of available components |
I guess this means that I miss the |
|
Yep - you can look in config.log to see why |
|
Great, thanks a lot! At least now I know where the issue comes from. I'm using an openmpi package built by my distribution (Arch), and my distribution does not officially support OFED. |
|
It seems now I got I can add that in my setup I'm able to use both |
|
The problem is right here: IB was unable to make the connection, and so openib disqualified itself. |
|
Specifically, I think you have two options here:
|
|
Thanks a lot for your help. If I tried to add the suggested flag (removing I would need to use MPI directly over ibverbs if possible... it this UCX adding an additional abstraction layer in the "datapath"? Do I need to build and install UCX and then re-configure and re-build openmpi so that it finds the new component? Thanks |
It does but UCX is actively maintained and optimized for infiniband hardware. Openib on the otherhand is old and on "maintenance only" stage. From our test results, UCX is generally better performing. (You can try to compare them yourself actually, just disable the rdmacm on openib btl as suggested.)
Yes, you do. |
|
Hi, thanks for the help. If you look at my latest output run, I actually tried to disable rdmacm, but with no luck. Anyway, I'll definitely try this UCX. I am interested in performance limits with very small messages (e.g. 8-16 bytes for message), that's why I want as little overhead/abstraction layer over the ibverbs. |
|
Hey @vmaffione sorry I didn't read your output log. So from what I'm seeing, the openib btl is initialized properly. So now the problem is not what you originally have anymore. The hang suggested that there is something wrong in the component selection. (as in which component to use to send data from node x to node y.) Can you try to use tcp to see if the application works? -mca btl tcp,self |
|
Yes, excluding rdmacm has effect, but it still hangs. This is weird, because I use rdmacm in another (non-MPI) program, with the very same setup (not at the same time, ofc). |
|
How many messages are you sending at the same time? Can you try openib btl with |
|
I'm sending one message at a time, but that's not point. My MPI program prints an "Hello world" after |
|
I added a printf() before and after |
|
Have you tried stripping the program of everything except the MPI_Init, print, and MPI_Finalize and see if that works. If it does, then it tells you that you truly are hanging on the communication, and the print statement after MPI_Init might be failing to show up for other reasons (like the daemons can't forward it because they are hung doing something else). I would then add --mca pmix_base_verbose 5 --mca grpcomm_base_verbose 5 to your cmd line and see if maybe the problem is that you are hanging in the fence operations |
|
Yes, right now my program is just a call to |
|
From gdb I see that there are two threads blocked on |
|
Yes - here is the problem: Looks like you are stuck in the fence. So try adding these to the cmd line (just to test this out): This tells OMPI not to execute a fence during MPI_Init |
|
Thanks. Unfortunately it seems to have no effect (unless the parameters are not spelled correctly, afaiu you get no feedback if they are misspelled). |
|
Hi, Any suggestions? |
|
seems like you have issue with ofed installation on your node. Does low-level (ibv_send_bw etc) tests working on your node? |
|
Yes, I am able to use ibv on my hardware. Both custom |
|
is there any error in dmesg? |
|
Nothing, unfortunately. Can it be a permission problem? |
|
Any ibv program or tool works correctly. Example: |
|
I was able to solve this openucx/ucx#2710 |
thx for pointing out |
|
I was never able to use openib. I switched to openucx and everything worked fine. You can follow the link posted in my last comment. |
|
er .. thx for the reply |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Background information
What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)
v3.1.0
Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)
Installed from precompiled distribution package.
Please describe the system on which you are running
Linux, Archlinux
x86_64, processor belonging to the Intel i7 family
Mellanox Technologies MT27500 Family [ConnectX-3]
Details of the problem
Hello,
I'm trying to run a simple (echo) MPI program over two hosts connected through a direct cable over Infiniband.
Each host has a 40 Gbps Mellanox card, with inifiniband support. Infiniband/ibverbs/RDMA software is provided by a
rdma-corepackage (https://github.com/linux-rdma/rdma-core).(The two hosts can also communicate on separate IP LAN).
The infiniband setup is already working: I'm able to run libibverbs programs over the Mellanox NICs.
However, when I try to run my MPI program forcing
openib, I get the following:It seems it fails to load
openib, but I don't see enough information to narrow down the issue.Any suggestions?
Thanks a lot,
Vincenzo
The text was updated successfully, but these errors were encountered: