memory leaks? #39

Closed
jtoy opened this Issue Feb 1, 2012 · 5 comments

4 participants

@jtoy

I've tracked down a memory leak to rsolr, where I normally create some threads to push data into solr. The ruby process uses up all the memory, but If I comment out rsolr so the while loop just pops items off the queue, the app stays at a constant 22 mb of use. This loop is used to push in millions of docs into solr.

25.times do |t|
  threads << Thread.new do
    rsolr = RSolr.connect :url => (CONFIG[$env]["solr"] ||'http://localhost:8983/solr')
    while true
      a = $q.pop
      if a
        begin
          rsolr.add a
          $inserts += 1
        rescue Exception => e
          puts e
          $ecount += 1
        end
      end
    end
  end
end
@mwmitchell

Hi,

Which line are you commenting out to prevent the leak? This could be RSolr, the XML builder lib or even Net::Http. Let me know if you have any more details.

  • Matt
@jtoy

Hi , please excuse the late reply, I just saw this. I tried a few different things. I made sure it wasn't my code by adding docs that only had a key of :id => i+1, and memory was still growing very fast. I then tried commenting out the code in RSolr::Client#update so that it didnt actually send data over http. That seems to have cut down the memory quite a bit, but it continues to rise in memory usage at a much slower rate.

@jtoy

I have a bare bones script that eats up the memory : https://gist.github.com/1844519 . If I make the s string very small, then you can barely tell its leaking memory. In my real app I have very large docs that could be MBs in size. Even while the main app is sleeping, its still seems to eat up memory with the threads adding docs.

@databyte

I originally thought I had a rsolr memory leak but I can't reproduce the memory leak just around rsolr calls. I'm widening my search to - well - everything.

Here's the adjusted script for memory allocation checks. It requires ruby 1.9.
https://gist.github.com/2597700

I left some of my test code commented out to play with things but running it over 2000 calls. I found no change in memory before and after the rsolr call (specifically solr.add. Especially if you force a GC.start.

@ndushay

1) the last comments imply rsolr prob. didn't have a memory leak
2) the last comment on this was almost 3 years ago - so unlikely to still be an issue
thus, i am closing this issue.

@ndushay ndushay closed this Feb 11, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment