Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Exit code for node tests is 0 when tests fails #227

Closed
c089 opened this Issue Jul 17, 2012 · 19 comments

Comments

Projects
None yet
4 participants
Contributor

c089 commented Jul 17, 2012

Running this test results in exit code being set to 0.

Owner

cjohansen commented Jul 18, 2012

Think I figured it out. I had the same problem in a project, and it happened because I used the globally installed buster binary while the project had buster also installed locally. Is that the case for you too?

Contributor

c089 commented Jul 18, 2012

Nope, using node_modules/buster/bin/buster-test also exits with code 0 on failing node test.

Contributor

c089 commented Jul 18, 2012

Actually, "node_modules/.bin/buster-test -e node" returns 1, only without the -e node it returns 0, so it might be related to #229.

Contributor

c089 commented Jul 18, 2012

Seems to be a combination of both:
buster-test: 0
node_modules/.bin/buster-test: 0
buster-test -e node: 0
node_modules/.bin/buster-test -e node: 1

Owner

cjohansen commented Jul 18, 2012

Thanks, I'll investigate.

@cjohansen cjohansen was assigned Aug 16, 2012

cowwoc commented Dec 21, 2012

Please note that node <test> is also returning 0 on failed tests.

Owner

dwittner commented Aug 7, 2013

@c089, could you please check, if the problem still exists (as you already did for #231?

Contributor

c089 commented Aug 9, 2013

current status: both buster-test -e node and buster-test return 0 when running the gist I posted in the original ticket.

Owner

dwittner commented Aug 9, 2013

Have you removed the statement var buster = require('buster'); from the configuration file before?
Having it set i also got a return code 0.

Contributor

c089 commented Aug 9, 2013

I now get 0 in any case: with and without the -e node, with and without the require('buster')

Owner

dwittner commented Aug 12, 2013

I cant't reproduce that behaviour yet. For your test i get a return code 1 without the require('buster') statement in the configuration file, for buster-test and buster-test -e node, on a linux and windows system. If i add the require('buster') statement to the configuration file, i get always a return code 0.

buster.js: 0.6.12 Beta 4
node.js: Windows: v0.10.0, Ubuntu: v0.10.15

Could you please provide more details about your installation/configuration and how you execute your tests (manually or via shell script for example)?

Contributor

c089 commented Aug 12, 2013

I used a locally installed buster, with no global one in the PATH (running node_modules/.bin/buster-test)

Owner

dwittner commented Aug 13, 2013

@c089, What is the value of the NODE_PATH?

More and more i think @cjohansen was already on the right way. I now installed a local version of Buster.JS in addition to a global version. If i run the global version by buster-test i can see, that at some point a switch is made from the global to the local version.

runConfigGroups
runConfig
loadRunner
loadRunner.callback
analyzer.run
node-runner.run
testRun.start
testRun.runTests
autoRun
autoRun.run 2
testRunner.create 2
foo: F
Failure: foo bar
    [assert.isTrue] Expected false to be true
    at Object.buster.testCase.bar (./fail.js:6:16)

1 test case, 1 test, 1 assertion, 1 failure, 0 errors, 0 timeouts
Finished in 0.05s

As you can see the switch is made at autoRun.run (all added log messages for the local version end with "2"). That switching finally leads to an exit code 0.

@c089, I assume that in your case a switch from the local to the global version is made.

Contributor

c089 commented Aug 13, 2013

I don't have a global version installed or any other version in the PATH. During the run, process.env.NODE_PATH is undefined.

Owner

dwittner commented Aug 13, 2013

@c089, Then i think @cjohansen was totally on the wrong way ;)

Can you please execute the following commands in your project folder (manually, not from within a shell script) and post the result?:

node -v
node_modules/.bin/buster -v
echo $NODE_PATH
node_modules/.bin/buster-test
echo $?

What OS do you use, Linux or OS X?

Contributor

c089 commented Aug 14, 2013

$ node -v
v0.10.15

$ node_modules/.bin/buster -v
Buster.JS version 0.6.12 Beta 4

$ echo $NODE_PATH

$ node_modules/.bin/buster-test
foo: F
Failure: foo bar
    [assert.isTrue] Expected false to be true
    at Object.buster.testCase.bar (./fail.js:6:16)

1 test case, 1 test, 1 assertion, 1 failure, 0 errors, 0 timeouts
Finished in 0.011s

$ echo $?
0
Owner

dwittner commented Aug 14, 2013

@c089, Strange. It seems i have the same configuration than you, but i can't reproduce the issue.

buster@ubuntu:~/issues/227$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 12.04.2 LTS
Release:    12.04
Codename:   precise

buster@ubuntu:~/issues/227$ node -v
v0.10.15

buster@ubuntu:~/issues/227$ ./node_modules/.bin/buster -v
Buster.JS version 0.6.12 Beta 4

buster@ubuntu:~/issues/227$ tree -a -L 2
.
├── buster.js
├── fail.js
└── node_modules
    ├── .bin
    └── buster

3 directories, 2 files

buster@ubuntu:~/issues/227$ cat buster.js
//var buster = require('buster');
var config = module.exports;
config['node tests'] = {
    environment: 'node',
    tests: [ 'fail.js' ]
};

buster@ubuntu:~/issues/227$ cat fail.js 
var buster = require('buster');
var assert = buster.assertions.assert;

buster.testCase('foo', {
    'bar': function() {
        assert.isTrue(false);
    }
});

buster@ubuntu:~/issues/227$ echo $NODE_PATH


buster@ubuntu:~/issues/227$ ./node_modules/.bin/buster-test
foo: F
Failure: foo bar
    [assert.isTrue] Expected false to be true
    at Object.buster.testCase.bar (./fail.js:6:16)

1 test case, 1 test, 1 assertion, 1 failure, 0 errors, 0 timeouts
Finished in 0.081s

buster@ubuntu:~/issues/227$ echo $?
1

As mentioned earlier, i have to remove the statement require('buster') from the configuration file to get an exit code 1.

Can you reproduce the issue on other projects and on other machines as well?

Contributor

c089 commented Aug 15, 2013

Doh, you're right - I tried again and it seems while testing the commands for my last comment, I cloned the gist again and forgot to remove the require statement again - I am now also getting the exit code 1. Sorry for that.

Owner

dwittner commented Aug 15, 2013

Never mind! That can happen. I am glad, that the problem is solved.

@dwittner dwittner closed this Aug 16, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment