Warden has a memory leak #9

Closed
lipengyuntian opened this Issue Mar 12, 2013 · 8 comments

Comments

Projects
None yet
5 participants

this problem can be very easily reproduced as follow steps:

  1. Start a java app in warden container
  2. Get information using command spawn cat /proc/stats and "Link cotainerid ", about every 3 seconds,
    you can find that warden's process memory increase from 23m to a very large number.

OS: Unbuntu 10.0
Warden version: current
Ruby's version is 1.92

Contributor

d commented Mar 17, 2013

Hi @lipengyuntian I cannot reproduce this. Does it make a difference if I swap out step 1 with an empty container?

thanks for your letter ,this operation will take a long time ,and if you start 30's instance ,this will happen more quickly.

Contributor

cf-frameworks commented Mar 19, 2013

Does this happen with HEAD as well? Which commit do you use?

(cc @pietern @cppforlife)

i guess fibers lead to this problem.

Contributor

pietern commented Mar 28, 2013

@lipengyuntian We can try to help if you give more information. Which version of warden (commit SHA) do you use when executing your tests?

i think this function below may cause memory increasing ,every request produces a fiber ,but fiber object may not be recoverd ,why not use a fiber pool
def receive_request(req = nil)
@requests << req if req

    # Don't start new request when old one hasn't finished, or the
    # connection is about to be closed.
    return if @blocked or @closing

    request = @requests.shift

    return if request.nil?

    logger.debug2(request)

    f = Fiber.new {
      begin
        @blocked = true
        @current_request = request
        process(request)

      ensure
        @current_request = nil
        @blocked = false

        if @draining
          logger.debug2("Finished processing request, closing")
          close
        else
          # Resume processing the input buffer
          ::EM.next_tick { receive_request }
        end
      end
    }

    f.resume
  end
Contributor

cf-frameworks commented Apr 9, 2013

Seems like you have a potential solution to this problem. Could you make a PR for this?

It is unclear how we can reproduce this problem, "using command spawn cat /proc/stats" is not possible, as spawn is not a command on the warden / dea machines, and /proc/stats is not a file.

The function #receive_request https://github.com/cloudfoundry/warden/blob/master/warden/lib/warden/server.rb#L397 is still implemented as noted but it is not clear that the use of Fiber is a problem.

Closing.

aramprice closed this Aug 7, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment