Investigate PhantomJS alternatives

[shanear]

Findings after a little bit of research

PhantomJS / Poltergeist / Sniffybara

• Already set up in Travis, and has automated 508 testing.
• Driving us crazy with the crashes

Capybara-webdriver

• Not a full browser
• Could have similar issues. It uses an old version of QT, to run just the webdriver.
• Difficult to set up. Have to intstall QT, then build the gem, which fails on build. Still need to explore if we decide to go this route
• Fast (according to testimony, since I still can't build the damn gem locally)

Selenium / Chrome

• Chose chrome instead of firefox because the driver seems better supported
• It's a real browser so,
  • Unlikely to have crashes
  • Tests will match reality much closer
• It was dead simple to set up, already have all the feature tests passing with chrome
• Looks easy to set up in Travis (Chromedriver support? travis-ci/travis-ci#272)
• Seems to be endorsed by protractor, the AngularJS testing framework
• It seems to be a little bit slower than PhantomJS (8% slower in some crude local testing)

Suggestion

Because it'd be such a pain to set up sniffy to work with the Selenium plugin, an
I'm leaning toward the Chrome fallback plan described here (https://blog.shakacode.com/yak-shaving-failing-integration-tests-with-react-a93444886c8c). We keep most of the benifits of PhantomJS, while mitigating the biggest drawback (the crashes). We could keep that going for a little while and see where to go from there.

Another easier option is to bump the retries to 10. I know its a little jank, but I don't see the downside (besides adding 10-15 seconds to some builds).

[joofsh]

why do you think it would be so hard to get sniffybara working with Selenium/Chrome? We're losing much more than 8% of our time atm dealing with test failures. I don't see any obvious benefits of keeping phantomJS in our stack. I'd personally prefer to get it entirely out of our stack.

But if it's truly easier to set up the chrome fallback, we can certainly try it first

[amprokop]

Another point. How well does Sniffybara work with SPAs? I can't remember it ever failing to be honest. I'd love to know a little bit of the history and where it's helped us in the past.

[amprokop]

Bumping the retries to 10 probably shaves off some of the problem but doesn't seem like a fix to me. I've observed failures even with 10 retries. It seems like phantom can get in a state where it just crashes over and over. No idea what's up with that.

[shanear]

@amprokop Yeah, they def work with react, and they are pretty important for our ability to deploy daily.

They caught a bunch of stuff with Reader (https://github.com/department-of-veterans-affairs/caseflow/pull/1379/files#r108937012)

@joofsh maybe it's not as bad as I think :/. The current implementation is pretty coupled to Poltergeist, and I don't really know how the Selenium capybara driver rolls.

[shanear]

@amprokop yeah that's so weird, sounds like retry 10 isn't really a good stop gap.

Will try and get the Chrome fallback solution within the week, and then we can look into going all Chrome (with sniffy converted over to Chrome)