Custom Launcher Errors in CI Mode #131

Closed
semperos opened this Issue Dec 17, 2012 · 11 comments

Comments

Projects
None yet
4 participants

I've configured a custom launcher as follows:

launchers:
  Node:
    command: "mocha -R tap test/*_test.js"
    protocol: "tap"
launch_in_dev:
  - Node
  - Chrome
launch_in_ci:
  - Node
  - Chrome

I've introduced code in my tests that causes a JavaScript ReferenceError. You can replicate the same with something as simple as f = new Foo() without defining Foo. When I run this with Mocha by itself, I see the stacktrace and Mocha exits with an exit code of 1, as expected. When I run the same Mocha command as a custom launcher via Testem, the stacktrace is not shown and Testem exits with an exit code of 0.

[ADDITION]
I forgot to add that this is happening in CI mode, not in dev mode.
[/ADDITION]

Can anyone shed light on why this is the case? This ends up silently swallowing errors that prevent my tests from even running, which is highly problematic.

Collaborator

airportyh commented Dec 17, 2012

I tried to setup the same scenario, but I am not able to reproduce - I do see the stacetrace. See gist https://gist.github.com/4318556

I just tried it out, and things got a bit weirder.

The setup in your Gist does work for me as well. However, if I mv error_test.js into a test folder and edit the Mocha command in the launcher to account for the placement of the test file:

mocha -R tap test/*_test.js

That does not work as expected in my setup. It swallows the error.

Thoughts?

Collaborator

airportyh commented Dec 17, 2012

Okay. I moved the file into the test folder. Let me show you my output

$ ls
test       testem.yml
$ testem ci
# Launching Node
# 
TAP version 13
not ok 1 - Node stuff should break should break
  ---
    message:    "ReferenceError: Foo is not defined"
    stacktrace: |
      "at Context.<anonymous> (/Users/airportyh/Home/Code/tm    p/testem_131/test/error_test.js:4:9)
      at Test.Runnable.run (    /usr/local/lib/node_modules/mocha/lib/runnable.    js:184:32)
      at Runner.runTest (    /usr/local/lib/node_modules/mocha/lib/runner.js:300:10    )
      at Runner.runTests.next (    /usr/local/lib/node_modules/mocha/lib/runner.js:346:12    )
      at next (/usr/local/lib/node_modules/mocha/lib/runner.    js:228:14)
      at Runner.hooks (    /usr/local/lib/node_modules/mocha/lib/runner.js:237:7)
      at next (/usr/local/lib/node_modules/mocha/lib/runner.    js:185:23)
      at Runner.hook (    /usr/local/lib/node_modules/mocha/lib/runner.js:205:5)
      at process.startup.processNextTick.process.    _tickCallback (node.js:244:9)"
  ...

# Launching Chrome
# 


1..1
# tests 1
# fail  1
$ 

The node launcher is working fine for me. Although for Chrome it doesn't work because the config file did not specific to look for the src_files in the sub-directory.

Can you paste your output if it deviates?

I put together a repo for my exact scenario: https://github.com/semperos/testing-testem

After running this simple example several times, it looks like there might be some kind of timing issue or race condition. Sometimes I get no error output, sometimes I get the stack trace.

Any ideas where timing issues might be, or how to expose them more consistently from a test run?

Collaborator

airportyh commented Dec 18, 2012

Interesting. I will dig deeper into it. Maybe a job for promises?

Collaborator

airportyh commented Dec 18, 2012

Just to make sure. What version of testem are you on?

0.2.50. I put my system specs in the README of that temp repo.

On Tuesday, December 18, 2012, Toby Ho wrote:

Just to make sure. What version of testem are you on?


Reply to this email directly or view it on GitHubhttps://github.com/airportyh/testem/issues/131#issuecomment-11474274.

Collaborator

airportyh commented Dec 18, 2012

I've tried and tried but couldn't reproduce it on my OSX Lion. Could be Mountain Lion specific maybe, or something else. Interesting thing is I could reproduce it on Windows. Just for giggles, would you try installing this fork of Mocha (my workaround for a Windows issue) and see if it fixes the issue for you?

https://github.com/airportyh/testem#known-issues

I'll have to try that fork of Mocha, but as another note, I get the same (bad) behavior on Ubuntu 12.10. My custom Node launcher just swallows the error, so the build looks like it passes and no stacktrace is output.

I think I'm encountering the same issue. If I remove the protocol: "tap" part of the custom launcher, it works as expected. If I leave it in, the entire test output is blank and passes.

$ cat testem.json 
{
  "launchers": {
    "custom": {
      "command": "exit 3",
      "protocol": "tap"
    }
  }
}

$ ./node_modules/.bin/testem ci -f testem.json -l custom
# Launching custom
# 


1..0
TAP version 13
# tests 0

# ok

Versions
testem => 0.2.83
node => 0.10.3

It looks like there's some mismatch between the tap reporter saying it's done (and successful) and the process exit handler saying there was an error.

Owner

johanneswuerbach commented Jan 2, 2016

This should be resolved in the latest testem version. You can install it using npm install -g testem@1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment