tests fail from clean environment

[towc]

Version

2.5.8

Reproduction link

https://github.com/vuejs/vue/blob/dev/.github/CONTRIBUTING.md (meta-issue)

Steps to reproduce

Opened up a fresh terminal and typed this in:

git clone https://github.com/vuejs/vue
cd vue
npm i
npm run setup
npm test

What is expected?

All tests should be passing. I might be confused on the development process, but the contribution file says "make sure npm test passes", under "Pull Request Guidelines", so I would expect npm test to pass on a fresh pull

What is actually happening?

This is the tail of npm test:

Finished in 25.953 secs / 25.514 secs @ 21:13:50 GMT 0100 (CET)

SUMMARY:
✔ 1056 tests completed
✖ 1 test failed

FAILED TESTS:
  Transition mode
    ✖ transition out-in on async component (resolve after leave complete)
      PhantomJS 2.1.1 (Linux 0.0.0)
    Expected 1 to be 0.
    webpack:///test/unit/features/transition/transition-mode.spec.js:480:44 <- index.js:159373:44
    shift@webpack:///test/helpers/wait-for-update.js:24:32 <- index.js:95787:36


=============================== Coverage summary ===============================
Statements   : 99.98% ( 4110/4111 )
Branches     : 94.67% ( 3340/3528 )
Functions    : 99.82% ( 568/569 )
Lines        : 99.98% ( 4004/4005 )
================================================================================
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! vue@2.5.8 test:cover: `karma start test/unit/karma.cover.config.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the vue@2.5.8 test:cover script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/user/.npm/_logs/2017-11-25T20_13_50_659Z-debug.log
npm ERR! Test failed.  See above for more details.

---

I doubt someone forgot to fix some code before AND the tools didn't pick it up AND the PR was accepted, so it's probably got to do with my environment. I'm running Ubuntu on i3wm, using fish shell, both of which might maybe not be considered in the test cases in some way. What should be done about this?

[madskonradsen]

One might also ask why Vue insists on using the kind of obsolete tool PhantomJS. It can't really keep up with the webstandards. Chrome headless is on the other hand up to date and much faster than PhantomJS (even though it's filesize is bigger (~71Mb Mac, ~90Mb Linux, ~110Mb Win for Chrome and 15-20 Mb for PhantomJS)) https://hackernoon.com/benchmark-headless-chrome-vs-phantomjs-e7f44c6956c

[eddyerburgh]

@madskonradsen I don't think Vue insists on using it, it's just that nobody has gotten around to adding a PR that switches to Chrome headless. If you're willing to make a PR, I'm sure we'd accept

[posva]

@eddyerburgh I tried recently in an unrelated PR but transitions tests must be adapted because they (6 of them) don't work with chrome headless 😆

[towc]

so if I wanted to submit a PR, do I just ignore that test that was failing?

[posva]

yes, they'll probably pass on CI

[yyx990803]

The transition test cases involves CSS transitions and timeouts, so they can be somewhat flaky depending on the machine they run on, but the failing case only fails once in a while, and passes most of the time (as seen on CI). They can be made more stable by increasing transition durations in the tests, but that would make tests run slower, so it's kind of a tradeoff. Another option is to completely refactor the way we test transitions to get rid of the flakiness, but that's in the backlog at the moment.

RE Phantom usage: We're intentionally staying on Phantom before we refactor the transition tests to work properly on Chrome headless.

[towc]

I think it would be a good idea to make a temporary note of it, in .github/CONTRIBUTING.md, to make it less confusing