severe performance degradation with jasmine runner in phantomjs

[rdy]

This is related to the issue we filed in the jasmine-gem: jasmine/jasmine-gem#225

When upgrading to Rails 4.1, @ohrite and I bumped jasmine, and found that our tests took an increasingly long time to run, measured in minutes (before it was in seconds). This only seems to occur running under phantomjs, its fine in the browser. After watching Activity Monitor, we determined that phantomjs leaks memory and starts thrashing after a few dozen examples.

We isolated the problem being introduced as part of this commit:
5f34be4

Before this commit everything is fine. Hope this helps!

[ragaskar]

hmm, I guess as a workaround for now you can use the selenium runner. I wonder if we don't track passing expectations if things improve? In many suites, that's going to be a lot of data; perhaps we can report it vs. accumulate it to reduce the number of objects in memory that we have references to.

[rdy]

Locking the jasmine core gem to 2.0.0 fixes the problem as well. It looks
like keeping track of the passed expectations is the problem in phantom.
I'm not sure why it's so slow. Our test suite contains about 600 tests.

On Jul 30, 2014, at 6:46 AM, Rajan Agaskar notifications@github.com wrote:

hmm, I guess as a workaround for now you can use the selenium runner. I
wonder if we don't track passing expectations if things improve? In many
suites, that's going to be a lot of data; perhaps we can report it vs.
accumulate it to reduce the number of objects in memory that we have
references to.

—
Reply to this email directly or view it on GitHub
#640 (comment).

[ragaskar]

My guess is if you had loads of failed expectations you'd see a similar slowdown.

[sheelc]

One point maybe worth mentioning is that after keeping track of passed expectations, the selenium_runner broke due to circular objects. That runner then had to be patched to just ignore the circular objects but phantomjs seemed to keep working without any changes.

Given that you are only seeing this slowdown with the phantom runner, I'm curious if it's related to a large number of circular objects. I haven't gotten around to testing this theory yet, but figured I'd mention it since it could be relevant?

[Gerg]

@tjarratt and I tried to reproduce this, but we were not able to find a test suite that performed worse after 5f34be4, including one with many circular objects. @rdy, do you have a representative example suite that can help us debug this issue?

The test suite we used for most of our investigation was based on large_test_suite_spec.js in jasmine-gem with an additional it for circular objects. We found it ran 60,000 specs in just over 2 minutes on our workstation. This suite has a lot of specs, is deeply nested but doesn't a lot of complicated expectations, spec logic, or any custom matchers.

[rdy]

I don't have access to the codebase that exhibited this problem but I believe some other folks are seeing the same issue. I'll see if I can get them to comment on this issue, and possibly provide a test case.

[infews]

@rdy where you guys loading TypeKit or GoogleFonts into the Jasmine runner HTML? There's a known issue with Phantom and these font libraries that slow down all tests - Capybara as well. I wonder if it's the same problem.

[rdy]

Interesting, but I don't think those fonts were being loaded, I don't have access to that test codebase right now so I can't really confirm.

We worked around the issue by using an earlier commit than one I referenced at the top of this commit, its really interesting that it only happens in phantom.

[neodude]

I'm running into this issue as well (independently from @rdy) - definitely not loading any webfonts in our jasmine runner. Just upgraded from 2.0.0 to 2.1.2, the CircleCI jasmine portion went from 30 seconds to 3 minutes. Same slowdown locally in PhantomJS, Chrome runs in 3 seconds. 889 specs. In terms of helpers we have jasmine-underscore and jasmine-jquery.

What can I do to help you guys debug this issue?

[infews]

Can you give us a super simple project that demonstrates the slowdown? Looking above, we've failed at finding such a case.

[neodude]

Got a repo case! Take a look at https://github.com/neodude/jasmine-performance-test, the salient file is large_test_suite_spec.js.

It's latest Rails (4.1.8) and jasmine-core (2.1.2). I adapted large_test_suite_spec.js, but instead of a simple true === true assertion, I asserted that two jQuery elements are not equal. Used latest jQuery too (2.1.1).

This suite has 500 passing specs, runs in ~0.3s on Chrome, and 1m46s in rake jasmine:ci.

My first guess would be the complex - possibly circular? - jQuery objects used in the expectation.

[neodude]

Pegged jasmine-core to this ref, and it totally fixed the performance issue in phantomjs. Thanks @slackersoft!