Memory leak in InternalThread::StartInternalThread()

[ryotat]

I think the following line

thread_ = new caffe::Thread
        (&InternalThread::InternalThreadEntry, this);

is causing a memory leak because this function is called every time BasePrefetchingDataLayer::Forward_cpu() is called and the allocated memory is not freed.

[jeffdonahue]

Good point -- thread_ should probably be changed from a caffe::Thread* to a shared_ptr<caffe::Thread>.

[ryotat]

Thanks for your reply. How about this line

  this->thread_ = new boost::thread(func, a1);

Isn't this also leaking? caffe::Thread does not seem to have a destructor. By the way, why are the member functions of caffe::Thread class in include/caffe/util/thread.hpp?

[jeffdonahue]

Yup, looks like a memory leak as well.

By the way, why are the member functions of caffe::Thread class in include/caffe/util/thread.hpp?

Not sure...this is my first time looking at all this code; seems it could use a good round of cleanups... (please feel free to send a PR).

[shelhamer]

@ryotat thanks for the close look and catching these issues! A fix is certainly in order, so a PR would be welcome!

[ryotat]

hmmm, I think I fixed this but I still have a problem that the amount of memory consumed by caffe increases linearly with the number of iterations. Is this a normal behavior? I was looking at the output of valgrind and found the above leak.

My issue might be related to this, but I am having the issue with both leveldb and LMDB.

[shelhamer]

@ryotat please PR your fix -- your cleanup is likely right but there is a recent issue with the data pipeline introduced in #1238 that we are fixing now.

[cvondrick]

After applying the fix suggested by ryotat, I still notice a leak on the latest master branch, supporting ryotat's observation that this may be a more complex leak. caffe memory usage increases on each iteration linearly, reaching tens of gigabytes and beyond.

The problem appears to be related to this line: https://github.com/BVLC/caffe/blob/master/src/caffe/layers/data_layer.cpp#L167. If I comment it out, memory remains mostly constant. However, I find it very difficult to believe that this is a bug is due to Google's protobuf library!

[ryotat]

Thanks for confirming my observation. It would be great if somebody with a better understanding of how protobuf / LMDB works could look into this.

[shelhamer]

Thread leak fixed by #1335.