Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Tests running very slow on Linux #41

Closed
rodrigotassinari opened this Issue Apr 28, 2011 · 38 comments

Comments

Projects
None yet

I'm using capybara-webkit 0.2.0 with RSpec 2.5.0 (in a Rails 3.0.7 app). Tests that take 13 seconds to run on OS X take 70 seconds to run on Ubuntu, either in a desktop with full X server or in a server with Xvfb. The hardware is very similar between all computers, and there's no noticeable difference in performance running other (not capybarazied) specs...

I've got no clue as to what may be causing this slowness... Anyone experiencing similar symptoms?

+1 I'm seeing very fast run times on OS X dev machines, but very slow test runs on our ubuntu CI server.

@ghost

ghost commented Oct 25, 2011

+1 seeing really slow running test on my brand new Thinkpad Core i7, running on Ubuntu 11.04 (now on 11.10).

Contributor

joemsak commented Nov 1, 2011

Just got it running on OS X here, and :js => true tests are running very slow, plus this error:

QTKitServer[14805:903] Error loading /Library/QuickTime/LiveType.component/Contents/MacOS/LiveType: dlopen(/Library/QuickTime/LiveType.component/Contents/MacOS/LiveType, 262): Library not loaded: /System/Library/PrivateFrameworks/LiveType.framework/Versions/A/LiveType
Referenced from: /Library/QuickTime/LiveType.component/Contents/MacOS/LiveType
Reason: image not found

I installed Qt from the packages mentioned in the Wiki

Contributor

joemsak commented Nov 1, 2011

Update:

Hiding google analytics inclusion behind <% if Rails.env.production? %> fixed my speed issue

We're seeing this same thing (with a CI server running Ubuntu 11.04).

I tried hiding Google Analytics in the environment my tests run in (based on joemsak's comment), but there was no noticeable speed-up.

cqiu commented Nov 15, 2011

We are having the same issue (our CI server is running Ubuntu 11.04) as well. And the solution of hiding google analytics didn't work for us either.

Contributor

joemsak commented Nov 15, 2011

Are there any other external JS you don't need for testing env? All we had to do was block those dependencies and we use a local jQuery for Rails.env.test? #=> true as well

cqiu commented Nov 15, 2011

we have separate env for test, dev and production. And in the test env, we don't use any external sources.

dpsk commented Jan 25, 2012

Did anyone find solution? Have the same issue.

Contributor

halogenandtoast commented Jan 27, 2012

Without a specific issue to address I'm going to suggest moving this conversation to the google group: https://groups.google.com/group/capybara-webkit

indrekj commented Feb 21, 2012

Anyone found solution? My tests are 3-5 times slower than with selenium.

+1, I'm hurting trying to optimize a suite where individual webkit tests are ~12 seconds! Going to group for discussion, see if we can narrow down options.

Same issue here!

My entire integration suite took ~ 13m on mac os x and ~ 1h30 on linux machines.

Tested against ubuntu desktop, ubuntu server, debian and centos.

chicks commented Mar 23, 2012

Is SELinux enabled?

@chicks nope:

[root@1 ~]# sestatus
SELinux status:                 disabled

From what I can see, there is a ~4-5 second delay when running the tests... It seems to come from Capybara::Driver::Webkit::Browser#pipe_readable? which spends the time waiting for a Timeout#timeout.

The semi-obvious looking line Timeout.timeout(5) do is not the cause of the delay. Although, yesterday my times just over 5 seconds, I thought I had found the cause...

I'm still trying to get to the bottom of this but would appreciate any help from anyone who knows the codebase.

Thread ID: 123693640
Total: 4.196852

 %self     total     self     wait    child    calls  name
  0.00      4.20     0.00     0.00     4.20        1  Capybara::Driver::Webkit::Browser#start_server (/home/anthony/.rvm/gems/ruby-1.9.3-p194@Edvance/gems/capybara-webkit-0.11.0/lib/capybara/driver/webkit/browser.rb:138}
  0.00      4.20     0.00     4.20     0.00        1  IO#eof? (ruby_runtime:0}
  0.00      4.20     0.00     0.00     4.20        1  Capybara::Driver::Webkit::Browser#forward_stdout (/home/anthony/.rvm/gems/ruby-1.9.3-p194@Edvance/gems/capybara-webkit-0.11.0/lib/capybara/driver/webkit/browser.rb:171}
  0.00      0.00     0.00     0.00     0.00        1  <Class::Thread>#current (ruby_runtime:0}
  0.00      4.20     0.00     0.00     4.20        1  Capybara::Driver::Webkit::Browser#pipe_readable? (/home/anthony/.rvm/gems/ruby-1.9.3-p194@Edvance/gems/capybara-webkit-0.11.0/lib/capybara/driver/webkit/browser.rb:191}
  0.00      0.00     0.00     0.00     0.00        1  Thread#abort_on_exception= (ruby_runtime:0}


Thread ID: 124092340
Total: 4.196780

 %self     total     self     wait    child    calls  name
  0.00      0.00     0.00     0.00     0.00        1  Kernel#sleep (ruby_runtime:0}
  0.00      4.20     0.00     4.20     0.00        1  Timeout#timeout (/home/anthony/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/timeout.rb:60}

mkoby commented Jun 4, 2012

I too am noticing this issue. We recently implemented Cucumber and use capybara-webkit as the driver. Works find on my MacBook Pro where the tests run in under 30 seconds. But on my Ubuntu box they take considerably longer, and the slow down seems to happen when the javascript tests are called.

The tests go pretty quick until I get to a javascript test and then they slow down and stay slowed down even after all the javascript tests have completed. So that might be a starting point.

I also want to note that I think it's poor form of the maintainers to close the issue when there is no known fix. Closing it out because there isn't a specific issue to address, to me, seems to kind of ignore the fact that an issue exists. In other words, you're hiding the issue, not actually fixing it. This issue is very real and obviously not resolved. I understand that the root cause of the issue isn't actually known at the moment, but saying it's "closed" is misleading.

mfoo commented Jun 7, 2012

I am also experiencing this problem, and running our cucumber features with capybara-webkit now takes around ten minutes for ~100 scenarios. On a Mac the tests run much faster and the webkit executable uses much more CPU. On Linux the webkit process uses very little CPU at all.

As with @mkoby, I'm also using javascript tests with the javascript annotation.

@anthonybatchelor-examtime what did you use to profile Capybara? I would like to see if I get similar results.

@mfoo I used ruby-prof and kcachegrind. There seems to be roughly 200ms stalls for every interaction with capybara-webkit. That's for things like does this element exist or fill some text in this box.

Contributor

benizi commented Jul 16, 2012

(Edit: see next comment for the simpler fix -- or the pull request -- #358)

I don't quite understand why this was marked as closed.

For me, the slowness appears to be caused by QtTcpServer. On my OpenSUSE machine, I found there was a delay of about 0.04 seconds per command being sent over TCP. (It looks like the poll() in webkit_server doesn't return instantly, despite there being data available.)

I've pushed a branch that uses QtLocalServer instead on the Qt side, and forces the use of UNIXSocket on the Ruby side. It's available as unix-socket on 4moms/capybara-webkit. In my horribly-slow test suite, the changed version is around 200x faster.

The way I've fixed it is incomplete in several ways:

  • It probably shouldn't force using UNIX sockets
  • Possibly there's some better/easier way to fix the QtTcpServer version
  • The test suite doesn't pass
  • The socket isn't properly cleaned up (requires rm /tmp/capybara-webkit after each run)

But, it's there if anyone wants to play with it.

Contributor

benizi commented Jul 16, 2012

As I suspected, there's an easier fix for the QtTcpServer version. Just needed to set TCP_NODELAY on the socket that gets created. Pushed a new branch (tcp-nodelay) with the fix.

antonio commented Jul 16, 2012

Wow, a 200x improvement is indeed impressive @benizi

Is this latest change stable, or does it still have the problems you commented before?

+1 to Antonio's comments - if it's fit for use (ie stable) as a drop in replacement, I'll switch to it immediately. Good work!

Contributor

benizi commented Jul 16, 2012

@antonio

Latest change should be stable. It doesn't use UNIX sockets, so doesn't have the same issues associated with my klunky first pass.

Taking "stable" in a different sense, though: I don't have green tests. (But I don't have green tests on master in the first place.)

antonio commented Jul 16, 2012

I'll give it a shot then. Thanks!

antonio commented Jul 16, 2012

Mmm I am afraid it's not working for me :(

Is there something I have to setup apart from pointing my Gemfile to a cloned version of your repo? First test fails with 'No response received from the server', while subsequent give 'Broken pipe'

Contributor

benizi commented Jul 16, 2012

Maybe you were using the original fix? (I switched branch names for the two fixes.) In my app with a slow test suite I changed this portion of my Gemfile from:

gem 'capybara-webkit'

to the following:

gem 'capybara-webkit',
  :git => 'https://github.com/4moms/capybara-webkit',
  :branch => 'tcp-nodelay'

Then after bundle update capybara-webkit it was much faster to run bundle exec rspec. Reverting the change and rerunning bundle update capybara-webkit, and it's back to multiple minutes instead of seconds.

I've got a different problem to Antonio testing this fix on Ubuntu 12.04, Ruby 1.9.3, Rails.

All my request tests are failing in the same way on visit some_path commands:

Failure/Error: visit new_school_year_path
     ArgumentError:
       wrong number of arguments(2 for 0)
     # ./spec/requests/school_years_spec.rb:39:in `block (4 levels) in <top (required)>'

Admittedly they fail very fast! ;-)

Update: Note that I followed Benizi's instructions above.

Update 2: Seems to be affecting only tests marked with ":js => true"

Contributor

benizi commented Jul 16, 2012

@davidkennedy-examtime I can't imagine changing the socket options would have that effect... maybe a result of also updating to the latest version of capybara-webkit? Or of updating something else in your Gemfile unintentionally?

More debug - I'd the following in my spec_helper.rband removing it has led to more tests passing - not all, but more.

  Capybara.register_driver :webkit do |app|
   Capybara::Driver::Webkit.new(app, :ignore_ssl_errors => true)
 end

Currently several tests just stall. I'm not sure what the cause is yet.

@benizi - I agree, it does seem unlikely, but I've only updated the one gem in isolation. Using the most recent vanilla gem works fine.

Contributor

benizi commented Jul 16, 2012

@davidkennedy-examtime - "most recent vanilla gem" = no :git option? Can you try the 'released-plus-tcp' branch I just pushed? It's master..tcp-nodelay cherry-picked onto the v0.12.1 tag.

gem 'capybara-webkit',
  :git => 'https://github.com/4moms/capybara-webkit',
  :branch => 'released-plus-tcp'

Clarification:

Editing Gemfile to be

   gem 'capybara-webkit'

followed by a bundle update capybara-webkit worked fine. This is the latest on RubyGems (0.12.1)

  gem 'capybara-webkit',
    :git => 'https://github.com/4moms/capybara-webkit'

followed by a bundle update capybara-webkit did not work. Note the lack of branch specifier. It fails on the tcp_nodelay branch too.

Editing Gemfile to be

  gem 'capybara-webkit',
    :git => 'https://github.com/thoughtbot/capybara-webkit'

followed by a bundle update capybara-webkit did not work. Looks like @benizi is right and there's some sort of upgrade bump I'm hitting?

And finally, editing Gemfile to be

gem 'capybara-webkit',
  :git => 'https://github.com/4moms/capybara-webkit',
  :branch => 'released-plus-tcp'

as requested above followed by a bundle update capybara-webkit ... I'm not sure! Failing tests, but they're failing on random places and not uniformly at the start as before. This might also be an upgrade problem, or related to how we use the library (we have a LOT of javascript). Investigating.

Update: Still looking into the fails (2 from 133 :js => true). The good news is that the whole suite was MUCH faster, ~9 mins to ~2 mins.

The fork from @benizi does function as a drop in replacement and gives a good improvement in speed (9m -> 6.5m, not the worlds nicest tests so this is a worst case) for webkit specific tests; the integration issues I had were timing problems (hidden divs, ajax calls etc) which had been masked by the overall slowness in the past.

I'm using the released-plus-tcp branch.

Update: For info, the particular speedup I'm seeing is 9m to 6.5m. However, these are very heavy tests so this probably represents a worst case. A couple of smaller tests are much improved.

antonio commented Jul 17, 2012

Sorry for not getting back to you before. I tried once again, this time with the released-plus-tcp branch. See the results below

Without the patch:

Finished in 6 minutes 47.43 seconds
191 examples, 0 failures

With it:

Finished in 4 minutes 23.02 seconds
191 examples, 0 failures

All the tests have the js flag.

I'll use the patch for a while to test its stability (when I tried yesterday, some tests were failing randomly with the Broken pipe error).

Thanks again

mfoo commented Aug 17, 2012

For anybody still looking into the speed that tests run at, as of August 10th this patch is in capybara-webkit master.

It knocked a 300 scenario, 4500 step, 45 minute test suite (almost all async js) down to about 16 minutes.

I also noticed that with ext4 there's a lot of journaling going on when mysql wrote to disk all the time during the tests, and by moving the mysql database to a ramdisk we got the time down to 6 minutes. If you don't use a ramdisk and instead use in-memory tables, this requires hacking the migration, limiting database field types, and not using transactions.

Ajedi32 commented Oct 31, 2012

I'm not sure whether or not this is the same issue, but I am using a version of capybara-webkit cloned directly from head and I am having a very similar problem on Windows.

Visiting a page (one step):

    user     system      total        real
1.719000   2.688000   4.407000 ( 44.955124)

Did you guys ever figure out what was going on with the "wrong number of arguments (2 for 0)" problem on the visit method? I'm having this exact same issue with the capybara-webkit gem, which goes away if I use the selenium driver instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment