Memory leak in v2.x only

[rolfv]

There was a memory leak reported by Steven Eliuk (s.eliuk@samsung.com) of Samsung running one of his applications. I managed to reproduce it using LAMMPS which I used before to show a memory leak. Here are the instructions on how to reproduce.

1. clone LAMMPS (git clone git://git.lammps.org/lammps-ro.git lammps)
2. cd lammps/src then compile with Open MPI of interest. To do this, set your path to Open MPI and type "make mpi"
3. run the example listed in lammps/examples/melt. To do this, first copy "lmp_mpi" from the src directory into the melt directory. Then you need to modify the in.melt file so that it will run for a while. Change "run 25" to "run 250000"
4. you can run by mpirun -np 2 lmp_mpi < in.melt 

I then run the following script to track memory consumption and we can see it rising.

#!/bin/bash                                                                                                                                     
PID=`pgrep lmp_mpi -d ,`
echo "Found pids=$PID"

while [ 1 ]
do ps -p ${PID} -o pid,comm,user,vsz
sleep 30
done

WIth master, 1.10.0 or 1.8.8 memory consumption is basically stable. With 2.x and you can see how it rises.

[rvandevaart@drossetti-ivy4 melt]$ ./chk-size.sh
Found pids=30412,30413
  PID COMMAND         USER        VSZ
30412 lmp_mpi         32341    373872
30413 lmp_mpi         32341    374676
  PID COMMAND         USER        VSZ
30412 lmp_mpi         32341    443156
30413 lmp_mpi         32341    443960
  PID COMMAND         USER        VSZ
30412 lmp_mpi         32341    512604
30413 lmp_mpi         32341    513412
  PID COMMAND         USER        VSZ
30412 lmp_mpi         32341    582052
30413 lmp_mpi         32341    583028
  PID COMMAND         USER        VSZ
30412 lmp_mpi         32341    651836
30413 lmp_mpi         32341    652644
  PID COMMAND         USER        VSZ
30412 lmp_mpi         32341    721620
30413 lmp_mpi         32341    722428
  PID COMMAND         USER        VSZ
30412 lmp_mpi         32341    791576
30413 lmp_mpi         32341    792380

[ggouaillardet]

@rolfv there are several issues here

the following patch fixes it if you do not use the openib btl

[gilles@motomachi ompi-v2.x]$ git diff
diff --git a/ompi/mca/pml/ob1/pml_ob1_isend.c b/ompi/mca/pml/ob1/pml_ob1_isend.c
index 2333b69..a96f683 100644
--- a/ompi/mca/pml/ob1/pml_ob1_isend.c
+++ b/ompi/mca/pml/ob1/pml_ob1_isend.c
@@ -220,7 +220,7 @@ int mca_pml_ob1_send(const void *buf,
         }
     }

-#if !OPAL_ENABLE_MULTI_THREADS
+#if !OMPI_ENABLE_MULTI_THREADS
     sendreq = mca_pml_ob1_sendreq;
     if( OPAL_UNLIKELY(NULL == sendreq) )
 #endif  /* !OPAL_ENABLE_MULTI_THREADS */
@@ -228,7 +228,7 @@ int mca_pml_ob1_send(const void *buf,
             MCA_PML_OB1_SEND_REQUEST_ALLOC(comm, dst, sendreq);
             if (NULL == sendreq)
                 return OMPI_ERR_TEMP_OUT_OF_RESOURCE;
-#if !OPAL_ENABLE_MULTI_THREADS
+#if !OMPI_ENABLE_MULTI_THREADS
             mca_pml_ob1_sendreq = sendreq;
 #endif  /* !OPAL_ENABLE_MULTI_THREADS */
         }
@@ -252,9 +252,12 @@ int mca_pml_ob1_send(const void *buf,
         ompi_request_wait_completion(&sendreq->req_send.req_base.req_ompi);

         rc = sendreq->req_send.req_base.req_ompi.req_status.MPI_ERROR;
-        MCA_PML_BASE_SEND_REQUEST_FINI(&sendreq->req_send);
     }
+#if OMPI_ENABLE_MULTI_THREADS
     OBJ_DESTRUCT(sendreq);
+#else
+    MCA_PML_BASE_SEND_REQUEST_FINI((&(sendreq)->req_send));
+#endif

     return rc;
 }

note things have changed a bit on master in order to handle the case of recursive sends,
so my best bet is we should unify v2.x and master when master is stable enough.

if you run with infiniband hardware, there is an other leak in the openib btl.
my first remark is we should not even be using the openib btl if running on a single node
(since sm and vader have a higher exclusivity than openib)

that being said, it is fine to run with --mca btl openib,self on a single node
and there is an other leak...

[hppritcha]

I think it might be better to push missing commits like 9a8a876
back to 2.x

---

sent from my smart phonr so no good type.

Howard
On Sep 30, 2015 11:01 PM, "Gilles Gouaillardet" notifications@github.com
wrote:

@rolfv https://github.com/rolfv there are several issues here

the following patch fixes it if you do not use the openib btl

[gilles@motomachi ompi-v2.x]$ git diff
diff --git a/ompi/mca/pml/ob1/pml_ob1_isend.c b/ompi/mca/pml/ob1/pml_ob1_isend.c
index 2333b69..a96f683 100644
--- a/ompi/mca/pml/ob1/pml_ob1_isend.c
+++ b/ompi/mca/pml/ob1/pml_ob1_isend.c
@@ -220,7 +220,7 @@ int mca_pml_ob1_send(const void *buf,
}
}

-#if !OPAL_ENABLE_MULTI_THREADS
+#if !OMPI_ENABLE_MULTI_THREADS
sendreq = mca_pml_ob1_sendreq;
if( OPAL_UNLIKELY(NULL == sendreq) )
#endif /* !OPAL_ENABLE_MULTI_THREADS /
@@ -228,7 +228,7 @@ int mca_pml_ob1_send(const void *buf,
MCA_PML_OB1_SEND_REQUEST_ALLOC(comm, dst, sendreq);
if (NULL == sendreq)
return OMPI_ERR_TEMP_OUT_OF_RESOURCE;
-#if !OPAL_ENABLE_MULTI_THREADS
+#if !OMPI_ENABLE_MULTI_THREADS
mca_pml_ob1_sendreq = sendreq;
#endif / !OPAL_ENABLE_MULTI_THREADS */
}
@@ -252,9 +252,12 @@ int mca_pml_ob1_send(const void *buf,
ompi_request_wait_completion(&sendreq->req_send.req_base.req_ompi);

     rc = sendreq->req_send.req_base.req_ompi.req_status.MPI_ERROR;

•    MCA_PML_BASE_SEND_REQUEST_FINI(&sendreq->req_send);
  

  }
  +#if OMPI_ENABLE_MULTI_THREADS
  OBJ_DESTRUCT(sendreq);
  +#else
• MCA_PML_BASE_SEND_REQUEST_FINI((&(sendreq)->req_send));
  +#endif

return rc;
}

note things have changed a bit on master in order to handle the case of
recursive sends,
so my best bet is we should unify v2.x and master when master is stable
enough.

if you run with infiniband hardware, there is an other leak in the openib
btl.
my first remark is we should not even be using the openib btl if running
on a single node
(since sm and vader have a higher exclusivity than openib)

that being said, it is fine to run with --mca btl openib,self on a single
node
and there is an other leak...

—
Reply to this email directly or view it on GitHub
#954 (comment).

[rolfv]

Thanks Gilles. I can confirm both your observations. I tried the patch and when running without openib, the memory leak is gone. When I do run with openib, there is still a memory leak.

In terms of your observation about using openib, yes, that is an issue that perhaps I should report. It has to do with some changes we made so that the openib is kept around because it supports RDMA. I have talked to Nathan about that.

[hppritcha]

Rolf could you cherry pick the missing commit i mentioned above and see if
that addresses the problem?

---

sent from my smart phonr so no good type.

Howard
On Oct 1, 2015 8:16 AM, "Rolf vandeVaart" notifications@github.com wrote:

Thanks Gilles. I can confirm both your observations. I tried the patch and
when running without openib, the memory leak is gone. When I do run with
openib, there is still a memory leak.

In terms of your observation about using openib, yes, that is an issue
that perhaps I should report. It has to do with some changes we made so
that the openib is kept around because it supports RDMA. I have talked to
Nathan about that.

—
Reply to this email directly or view it on GitHub
#954 (comment).

[rolfv]

I just tested the cherry pick you mentioned - 9a8a876 - and it did not fix the memory leak when running on single node with self,sm. Not sure why, but it does not address the problem.

[hppritcha]

what's the magic make line to build lamps? I don't seem to be smart enough to figure this one out.

[ggouaillardet]

cd src
make mpi

(see Rolf initial message, and scroll right since the lines are pretty long)

[hppritcha]

thanks, I thought I had to use one of the many makefiles in MAKE.

[hppritcha]

I ran using v2.x with open-mpi/ompi-release#625 included using both self,vader and self, sm and was not able to reproduce the lammps memory leak.

@rolfv could you ask Steve to try again with a nightly snapshot of v2.x?

[rolfv]

I just retested and I agree that the memory leak is gone. We do not need Steve to run anything as the lammps reproducer is sufficient. So in my opinion, we can close this as fixed. Sound good to you @hppritcha ?