Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using multiple reporters results into wrong exit code if test fails #804

Closed
teemuhirsikangas opened this issue Jan 26, 2018 · 5 comments
Closed
Assignees
Labels
bug
Milestone

Comments

@teemuhirsikangas
Copy link

@teemuhirsikangas teemuhirsikangas commented Jan 26, 2018

Summary:
Using multiple reporters with Lab v.15.2.0 -> wrong exit code when test fails
Using single reporter with Lab v.15.2.0 -> test doesn't run
Using no reporters -> test fail with correct exit code -> OK

See below for detailed logs:

This works ok with Lab version 14.3.2:

unit.js:

const { expect } = require('code');
const { it } = exports.lab = require('lab').script();

it('returns true when 1 + 1 equals 2', () => {

  expect(1 + 1).to.equal(3);
});

lab --verbose unit.js -r console -o stdout -r html -o coverage.html -r lcov -o lcov.info -r json -o data.json

✖ 1) returns true when 1 + 1 equals 2

Failed tests:

  1) returns true when 1 + 1 equals 2:
      actual expected
      23
      Expected 2 to equal specified value: 3
      at it (/unit.js:6:122

1 of 1 tests failed
Test duration: 8 ms
Assertions count: 0 (verbosity: 0.00)
No global variable leaks detected
Coverage: 100.00%

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! unit@8.0.0 test1: `lab --verbose unit.js -r console -o stdout -r html -o coverage.html -r lcov -o lcov.info -r json -o data.json`
npm ERR! Exit status 1

But when using latest 15.2.0 version it does not work as expected and gives wrong exit code:

lab --verbose unit.js -r console -o stdout -r html -o coverage.html -r lcov -o lcov.info -r json -o data.json

✖ 1) returns true when 1 + 1 equals 2

Failed tests:

  1) returns true when 1 + 1 equals 2:

      actual expected
      23
      Expected 2 to equal specified value: 3
      at it (/unit.js:6:122

1 of 1 tests failed
Test duration: 5 ms
No global variable leaks detected
Coverage: 100.00%


Also having just one reporter it doesn't even run the tests:

lab --verbose unit.js -r console -o console.log

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! unit@8.0.0 test1: `lab --verbose unit.js -r console -o console.log`
npm ERR! Exit status 1

0 info it worked if it ends with ok
1 verbose cli [ '/usr/bin/node', '/usr/bin/npm', 'run', 'test1' ]
2 info using npm@5.6.0
3 info using node@v8.9.4
4 verbose run-script [ 'pretest1', 'test1', 'posttest1' ]
5 info lifecycle unit@8.0.0~pretest1: unit@8.0.0
6 info lifecycle unit@8.0.0~test1: unit@8.0.0
7 verbose lifecycle unit@8.0.0~test1: unsafe-perm in lifecycle true
8 verbose lifecycle unit@8.0.0~test1: PATH: /usr/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/home/username/unit/node_modules/.bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
9 verbose lifecycle unit@8.0.0~test1: CWD: /home/username/unit
10 silly lifecycle unit@8.0.0~test1: Args: [ '-c', 'lab --verbose unit.js -r console -o console.log' ]
11 silly lifecycle unit@8.0.0~test1: Returned: code: 1  signal: null
12 info lifecycle unit@8.0.0~test1: Failed to exec test1 script
13 verbose stack Error: unit@8.0.0 test1: `lab --verbose unit.js -r console -o console.log`
13 verbose stack Exit status 1
13 verbose stack     at EventEmitter.<anonymous> (/usr/lib/node_modules/npm/node_modules/npm-lifecycle/index.js:285:16)
13 verbose stack     at emitTwo (events.js:126:13)
13 verbose stack     at EventEmitter.emit (events.js:214:7)
13 verbose stack     at ChildProcess.<anonymous> (/usr/lib/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
13 verbose stack     at emitTwo (events.js:126:13)
13 verbose stack     at ChildProcess.emit (events.js:214:7)
13 verbose stack     at maybeClose (internal/child_process.js:925:16)
13 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)
14 verbose pkgid unit@8.0.0
15 verbose cwd /home/username/unit
16 verbose Linux 4.9.0-5-amd64
17 verbose argv "/usr/bin/node" "/usr/bin/npm" "run" "test1"
18 verbose node v8.9.4
19 verbose npm  v5.6.0
20 error code ELIFECYCLE
21 error errno 1
22 error unit@8.0.0 test1: `lab --verbose unit.js -r console -o console.log`
22 error Exit status 1
23 error Failed at the unit@8.0.0 test1 script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 1, true ]


And if I don't give any reporters, it works just fine

lab --verbose unit.js

✖ 1) returns true when 1 + 1 equals 2

Failed tests:

  1) returns true when 1 + 1 equals 2:
      actual expected
      23
      Expected 2 to equal specified value: 3
      at it (/unit.js:6:20

1 of 1 tests failed
Test duration: 7 ms
No global variable leaks detected

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! unit@8.0.0 test1: `lab --verbose unit.js`
npm ERR! Exit status 1

Please let me know if you need further information

@geek geek added the bug label Jan 26, 2018
@geek

This comment has been minimized.

Copy link
Member

@geek geek commented Jan 26, 2018

@teemuhirsikangas thanks for reporting and providing all of the information, I'll take a look at this later today.

@geek

This comment has been minimized.

Copy link
Member

@geek geek commented Jan 29, 2018

@teemuhirsikangas in #806 I added a couple of tests for the cases you included, but am unable to reproduce. What is the exit code you are seeing? (echo $?)

@geek geek added non issue and removed bug labels Jan 29, 2018
@geek geek self-assigned this Jan 29, 2018
@geek geek closed this in #806 Jan 29, 2018
@teemuhirsikangas

This comment has been minimized.

Copy link
Author

@teemuhirsikangas teemuhirsikangas commented Jan 30, 2018

@geek The exit code was 0 when tests was failing with multiple reporters.
I made small project with script where this reproduces for me every time:

https://github.com/teemuhirsikangas/lab-testing

Just run:
npm run test test fail but exit code is 0
npm run test1 test fail and exit code is 1 -> OK
npm run test2 does not run test, and exit code is 1

@geek geek reopened this Jan 30, 2018
@geek geek added bug and removed non issue labels Jan 30, 2018
@geek geek added this to the 15.2.1 milestone Jan 30, 2018
@geek

This comment has been minimized.

Copy link
Member

@geek geek commented Jan 30, 2018

@teemuhirsikangas thanks so much for the help, I have a fix ready that will go out with the next release.

@geek geek closed this in #807 Jan 30, 2018
@teemuhirsikangas

This comment has been minimized.

Copy link
Author

@teemuhirsikangas teemuhirsikangas commented Jan 31, 2018

no problem, happy to help, and the fix works great 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.