Skip to content

Possibility to restart webkit server #443

Closed
wants to merge 1 commit into from

9 participants

@simonoff
simonoff commented Jan 3, 2013

No description provided.

@mhoran
mhoran commented Jan 4, 2013

Instead of killing the entire server, I'm more inclined to figure out why Session#reset! isn't cleaning up resources. Calling Session#reset! should delete all active WebPage instances, which in turn frees the NetworkAccessManager and all pending and finished QNetworkReply objects. If there's a leak somewhere, I'd rather clean that up, since killing the server is really just a cover up for a memory leak.

@simonoff
simonoff commented Jan 4, 2013

I'm not familiar in C++ so I cannot investigate why Session#reset! is not working. But memory leak is available when you using Webkit for scrapping instead testing purposes.

@mhoran
mhoran commented Jan 5, 2013

I'd recommend manually calling Session#reset! where you were planning to restart webkit_server. This should have the same effect, and is what normally happens between test runs with rspec and Cucumber.

@simonoff
simonoff commented Jan 5, 2013

Session#reset! not working! It not reset webkit browser. I have a some scrapper what uses capybara-webkit and after 2000 requests browser uses 2Gb of memory. Session#reset! don't reset browser and it still using 2Gb of memory. So for me is better kill webkit_server and run it again.

@seangeo
seangeo commented Jan 11, 2013

I've experienced similar behaviour on stillalive.com test runners. These were very long running processes that hit multiple websites running various scripts. I've found that restarting webkit_server between test runs gives the best overall stability. I have a hunch that webkit caching could be the culprit here but haven't investigated further, it's probably worth fixing though, because I think long running stability and webkit_server restarts are both valid use cases.

Having said that, this implementation should probably provide a destroy method on the browser or driver object, so you don't have to do session.driver.browser.connection.restart. In fact, the connection isn't publicly available on the browser object anyway... so unless I'm mistaken you'd have to hold onto a reference to the connection when the browser was created in order to reset it. So maybe the api needs some work here. I also think destroy is preferable to restart, since it makes sense to destroy and recreate the entire session object to avoid any possibility of left over state.

I have some mixins which I use internally to add the destroy behaviour to Browser which I could turn into a PR if you're interested.

@ezkl
ezkl commented Jan 15, 2013

@seangeo I'd be very interested in seeing the mixins you're using.

@jobseekerltd

There is a memory leak in WebPageManager.cpp:

WebPageManager::WebPageManager(QObject *parent) : QObject(parent) {
  m_ignoreSslErrors = false;
  m_cookieJar = new NetworkCookieJar(this);
  m_success = true;
  m_loggingEnabled = false;
  m_ignoredOutput = new QString();
  m_timeout = -1;
  createPage(this)->setFocus();
}

Debug messages will be appended m_ingoredOuput and it will keep growing infinitely as long as webkit server is running. I changed m_ignoredOutput to QFile and it reduces memory usage significantly for us:

  m_ignoredOutput = new QFile(this);

Unfortunately the our code lives in a private repo so I can't create a pull request. I hope someone can incorporate this into master branch.

@mhoran
mhoran commented Jan 15, 2013

Great find @climbingrose. I'll commit that change tomorrow.

@mhoran
mhoran commented Jan 16, 2013

I saw a 10MB decrease in memory usage while running the capybara-webkit tests with the change @climbingrose proposed. I've committed the change to master. It'd probably make sense not to instantiate a QDebug object every time the logger is called as well, but this is a great improvement already.

@zhengjia

With master branch I am still seeing increasing memory usage on webkit_server process. It grew about 300mb in about half an hour. I agree with @seangeo there should be a destroy method on the browser object. Selenium has a similar method browser.quit that can close and start a new browser window.

@mhoran
mhoran commented Jul 2, 2014

Closing this out for now. There's still no upstream API for this and I'd prefer to fix any memory leaks instead of implementing this as a workaround. If it really becomes an issue for folks I'm willing to reconsider. However even with large test suites on hosted CI instances I haven't run up against blocking memory issues.

@mhoran mhoran closed this Jul 2, 2014
@jarthod
jarthod commented Apr 4, 2016

Sorry for that but I'll have to reopen this ☺

We have a 14k tests suite with about 1k integration tests, using capybara-webkit and selenium. And when run on our CI the webkit-server process takes up to 5GB of memory (growing pretty much linearly with test count) whereas doing the same thing with Firefox (selenium) never go over 300MB.

This is, as you can guess, problematic for parallel test execution, even on a big machine.

I tried updating capybara-webkit (now using 1.9.0) and Qt (now using 5.2.1) but it didn't change anything. So I guess there are memory leaks of course but although I'm comfortable with C/C++/Valgrind I'm not sure how to help here. Do you have any way to extract information from my runs in a way that could be actionable to you ?

In the meanwhile I'll try hacking around to get the webkit-server to restart during the test run as it's the only way I have to save my CI.

@jferris
thoughtbot, inc. member
jferris commented Apr 6, 2016

@jarthod I just opened #898 to provide better separation of concerns for booting/connecting to the server. You can use that branch to pass your own server that's already running via Valgrind:

% valgrind /path/to/capybara-webkit/bin/webkit_server
==4882== Memcheck, a memory error detector
==4882== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==4882== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==4882== Command: src/webkit_server
==4882== 
Capybara-webkit server started, listening on port: 43065
class RunningServer
  attr_reader :port

  def initialize(port)
    @port = port
  end

  def start
  end
end

require "capybara/webkit"

Capybara.register_driver :running_webkit do |app|
  webkit_server = RunningServer.new(43065)
  webkit_connection = Capybara::Webkit::Connection.new(server: webkit_server)
  webkit_browser = Capybara::Webkit::Browser.new(webkit_connection)

  Capybara::Webkit::Driver.new(app, browser: webkit_browser)
end

Capybara.driver = :running_webkit

You can then boot your server process with valgrind, gbd, or anything else that helps you find leaks.

If you get any interesting heap output from Valgrind, feel free to include it here.

@jarthod
jarthod commented Apr 6, 2016

Ok great, I'll have a look at this. For the record and for other people looking at this thread, here is what I managed to hack in order to restart the webkit server every 20 specs (to minimize bootup time impact):

RSpec.configure do |config|

  # ...

  $webkit_server_age = 0
  config.after :each, type: :feature do |example|
    if [:selenium, :webkit].include?(Capybara.current_driver)
      Capybara.reset!
    end
    if Capybara.current_driver == :webkit
      # Restart webkit_server every 10 spec to reduce memory leak
      if $webkit_server_age > 20
        driver = Capybara.current_session.driver
        conn = driver.instance_variable_get('@browser').instance_variable_get('@connection')
        Process.kill("TERM", conn.send(:discover_pid))
        conn.restart
        driver.reset!
        $webkit_server_age = 0
      else
        $webkit_server_age += 1
      end
    end
  end

This significantly limited the memory leak, the webkit_server never goes above 500MB now. Of course it's a crappy solution but it's the only one I found and even though I would love to help reducing these memory leaks I know it's gonna be hard and we'll never eliminate them all.

@jferris
thoughtbot, inc. member
jferris commented Apr 6, 2016

@jarthod The Connection class has pid method that returns the PID, which would let you remove one private method call.

@jarthod
jarthod commented Apr 7, 2016

Good point, thanks.

@jarthod
jarthod commented Apr 7, 2016

Ok after a first round of investigation, I found that the memory leak is only present on some of our specs, loading a specific page. I tried removing the JS code specific to this page and the leak seems to be gone so it's at least liked to what this JS do. Unfortunately there's a log of JS here so I'm digging and will keep you posted.

For the record here is what I used to track down which specs caused leaks:

def RSS pid
  `ps -o rss -p #{pid}`.strip.split.last.to_i / 1024
end

RSpec.configure do |config|

  # ...

  $webkit_rss = 0
  config.after :each, type: :feature do |example|
    if Capybara.current_driver == :webkit
      driver = Capybara.current_session.driver
      conn = driver.instance_variable_get('@browser').instance_variable_get('@connection')
      mem = RSS(conn.pid)
      puts "webkit_server mem = %4d MB (was %4d MB, %+3d MB)" % [mem, $webkit_rss, (mem - $webkit_rss)]
      $webkit_rss = mem
    end
  end
end
@jarthod
jarthod commented Apr 7, 2016

Aha I think I found the culprit, still not sure why it leaks yet. It seems to come from this line:

var favico = new Favico({animation: 'none'});

We're using favico.js to add badges to the page favicon in realtime. And it seems that each time we instantiate this, the webkit_server memory increase between 5 and 20MB, and these are never freed. Instantiating / updating multiple time in the same page doesn't seem to increase memory more.

So here are some of my ideas:

  • Is there something like a favicon cache ? Is there any way to purge it ?
  • Maybe it's the canvas element creation which leaks ?

Any thoughts?


Here is a log of 20 tests just doing visit '/path' with favico.js:

webkit_server mem =  123 MB (was    0 MB, +123 MB)
webkit_server mem =  128 MB (was  123 MB,  +5 MB)
webkit_server mem =  136 MB (was  128 MB,  +8 MB)
webkit_server mem =  153 MB (was  136 MB, +17 MB)
webkit_server mem =  161 MB (was  153 MB,  +8 MB)
webkit_server mem =  172 MB (was  161 MB, +11 MB)
webkit_server mem =  179 MB (was  172 MB,  +7 MB)
webkit_server mem =  190 MB (was  179 MB, +11 MB)
webkit_server mem =  197 MB (was  190 MB,  +7 MB)
webkit_server mem =  208 MB (was  197 MB, +11 MB)
webkit_server mem =  220 MB (was  208 MB, +12 MB)
webkit_server mem =  228 MB (was  220 MB,  +8 MB)
webkit_server mem =  235 MB (was  228 MB,  +7 MB)
webkit_server mem =  250 MB (was  235 MB, +15 MB)
webkit_server mem =  263 MB (was  250 MB, +13 MB)
webkit_server mem =  271 MB (was  263 MB,  +8 MB)
webkit_server mem =  275 MB (was  271 MB,  +4 MB)
webkit_server mem =  287 MB (was  275 MB, +12 MB)
webkit_server mem =  299 MB (was  287 MB, +12 MB)
webkit_server mem =  318 MB (was  299 MB, +19 MB)

And here is the same log without favico.js

webkit_server mem =  123 MB (was    0 MB, +123 MB)
webkit_server mem =  134 MB (was  123 MB, +11 MB)
webkit_server mem =  141 MB (was  134 MB,  +7 MB)
webkit_server mem =  146 MB (was  141 MB,  +5 MB)
webkit_server mem =  146 MB (was  146 MB,  +0 MB)
webkit_server mem =  146 MB (was  146 MB,  +0 MB)
webkit_server mem =  147 MB (was  146 MB,  +1 MB)
webkit_server mem =  157 MB (was  147 MB, +10 MB)
webkit_server mem =  162 MB (was  157 MB,  +5 MB)
webkit_server mem =  163 MB (was  162 MB,  +1 MB)
webkit_server mem =  161 MB (was  163 MB,  -2 MB)
webkit_server mem =  161 MB (was  161 MB,  +0 MB)
webkit_server mem =  161 MB (was  161 MB,  +0 MB)
webkit_server mem =  161 MB (was  161 MB,  +0 MB)
webkit_server mem =  162 MB (was  161 MB,  +1 MB)
webkit_server mem =  163 MB (was  162 MB,  +1 MB)
webkit_server mem =  161 MB (was  163 MB,  -2 MB)
webkit_server mem =  158 MB (was  161 MB,  -3 MB)
webkit_server mem =  157 MB (was  158 MB,  -1 MB)
webkit_server mem =  158 MB (was  157 MB,  +1 MB)
@twalpole
twalpole commented Apr 7, 2016

@jarthod There's an open issue on the favico.js project about a memory leak - might be what you're seeing: ejci/favico.js#90

@jarthod
jarthod commented Apr 7, 2016

It's probably the cause, yes, but in this case shouldn't the page change + Capybara.reset! trash this lost memory ? like closing + opening a new tab ? Because if js leaks bloat the memory after the page is closed that's bad.

@mhoran
mhoran commented Apr 8, 2016

It depends on where the leak is. We don't completely tear down the webkit_server on reset!, so it's likely that there's some memory sticking around outside of the QWebPage object that is recreated. If this is a known issue for other WebKit based browsers, it's likely there's an issue deep down in WebKit/WebCore that is not cleaned up without a restart of the server.

@jarthod jarthod referenced this pull request in ejci/favico.js Apr 9, 2016
Open

Detached DOM Nodes #90

@jarthod
jarthod commented Apr 9, 2016

Ok thanks :/ I'm gonna help fixing this leak then, and pray for all our js and libs not to leak like that again ☺

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.