Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

memory leaks? #39

Closed
jtoy opened this Issue · 5 comments

4 participants

jtoy Matt Mitchell David Sommers Naomi Dushay
jtoy

I've tracked down a memory leak to rsolr, where I normally create some threads to push data into solr. The ruby process uses up all the memory, but If I comment out rsolr so the while loop just pops items off the queue, the app stays at a constant 22 mb of use. This loop is used to push in millions of docs into solr.

25.times do |t|
  threads << Thread.new do
    rsolr = RSolr.connect :url => (CONFIG[$env]["solr"] ||'http://localhost:8983/solr')
    while true
      a = $q.pop
      if a
        begin
          rsolr.add a
          $inserts += 1
        rescue Exception => e
          puts e
          $ecount += 1
        end
      end
    end
  end
end
Matt Mitchell
Owner

Hi,

Which line are you commenting out to prevent the leak? This could be RSolr, the XML builder lib or even Net::Http. Let me know if you have any more details.

  • Matt
jtoy

Hi , please excuse the late reply, I just saw this. I tried a few different things. I made sure it wasn't my code by adding docs that only had a key of :id => i+1, and memory was still growing very fast. I then tried commenting out the code in RSolr::Client#update so that it didnt actually send data over http. That seems to have cut down the memory quite a bit, but it continues to rise in memory usage at a much slower rate.

jtoy

I have a bare bones script that eats up the memory : https://gist.github.com/1844519 . If I make the s string very small, then you can barely tell its leaking memory. In my real app I have very large docs that could be MBs in size. Even while the main app is sleeping, its still seems to eat up memory with the threads adding docs.

David Sommers

I originally thought I had a rsolr memory leak but I can't reproduce the memory leak just around rsolr calls. I'm widening my search to - well - everything.

Here's the adjusted script for memory allocation checks. It requires ruby 1.9.
https://gist.github.com/2597700

I left some of my test code commented out to play with things but running it over 2000 calls. I found no change in memory before and after the rsolr call (specifically solr.add. Especially if you force a GC.start.

Naomi Dushay
Owner

1) the last comments imply rsolr prob. didn't have a memory leak
2) the last comment on this was almost 3 years ago - so unlikely to still be an issue
thus, i am closing this issue.

Naomi Dushay ndushay closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.