I've tracked down a memory leak to rsolr, where I normally create some threads to push data into solr. The ruby process uses up all the memory, but If I comment out rsolr so the while loop just pops items off the queue, the app stays at a constant 22 mb of use. This loop is used to push in millions of docs into solr.
25.times do |t|
threads << Thread.new do
rsolr = RSolr.connect :url => (CONFIG[$env]["solr"] ||'http://localhost:8983/solr')
a = $q.pop
$inserts += 1
rescue Exception => e
$ecount += 1
Which line are you commenting out to prevent the leak? This could be RSolr, the XML builder lib or even Net::Http. Let me know if you have any more details.
Hi , please excuse the late reply, I just saw this. I tried a few different things. I made sure it wasn't my code by adding docs that only had a key of :id => i+1, and memory was still growing very fast. I then tried commenting out the code in RSolr::Client#update so that it didnt actually send data over http. That seems to have cut down the memory quite a bit, but it continues to rise in memory usage at a much slower rate.
I have a bare bones script that eats up the memory : https://gist.github.com/1844519 . If I make the s string very small, then you can barely tell its leaking memory. In my real app I have very large docs that could be MBs in size. Even while the main app is sleeping, its still seems to eat up memory with the threads adding docs.
I originally thought I had a rsolr memory leak but I can't reproduce the memory leak just around rsolr calls. I'm widening my search to - well - everything.
Here's the adjusted script for memory allocation checks. It requires ruby 1.9.
I left some of my test code commented out to play with things but running it over 2000 calls. I found no change in memory before and after the rsolr call (specifically solr.add. Especially if you force a GC.start.
1) the last comments imply rsolr prob. didn't have a memory leak
2) the last comment on this was almost 3 years ago - so unlikely to still be an issue
thus, i am closing this issue.