Wallaby.js Tests output not reporting filename in VS Code

[Sequoia]

Issue description or question

I'd like to be able to jump to the failing test by clicking on it.

Wallaby.js configuration file

module.exports = function () {
  return {
    files: [
      {pattern: '\.env', instrument: false },
      'package.json',
      'data/**/*.json',
      'server/**/*.js',
      'server/**/*.json',
      'common/**/*.js',
      'common/**/*.json',
      'test/setup/*.js',
      'test/data/**/*.json',
      'test/data/**/*.xml',
      'test/fixtures/**/*.js',
      'test/mocha.opts'
    ],

    tests: [
      'test/**/*.spec.js'
    ],

    testFramework : 'mocha',

    env: {
      type: 'node',
      params: {
        env: 'NODE_ENV=TEST'
      }
    },

    workers: {
      recycle: true
    },

    // setup: function(wallaby){
    //   var mocha = wallaby.testFramework;
    //   mocha.setup({'recursive' : true});
    // }
  };
};

Code editor or IDE name and version

Visual Studio Code Version 1.2.1 (1.2.1)

OS name and version

OSX 10.11.5 (15F34)

Test output:

​4 failing tests, 557 passing
​ 
​ Send Digest Emails getSubscribedClients includes recently emailed when explicitly requested [1 ms]​
​  
​  limit value must be non-negative
​ 
​ Send Digest Emails getSubscribedClients omits recently emailed by default [2 ms]​
​  
​  limit value must be non-negative
​ 
​ Send Digest Emails getSubscribedClients gets clients that need sending [3 ms]​
​  
​  limit value must be non-negative
​ 
​ Send Digest Emails getSubscribedClients gets only subscribed clients [4 ms]​
​  
​  limit value must be non-negative

I have no way to get to the files besides "find in project" on part of the test description, but that's no better than just running in terminal.

[Sequoia]

hmmm it's even worse on errors:

​0 failing tests, 481 passing
​
​"before all" hook: done() invoked with non-Error: Error while attempting search - HTTP Status 500 returned (Internal Server Error)

👆 this is bad :(

[ArtemGovorov]

Wallaby.js actually supports a couple of powerful navigation commands allowing you to jump to a failing test from anywhere. Check out the "Jump to" commands, when invoked from the line of code it will let you jump to the failing test that covers the line (or to the first failing test if the code is not covered by a failing test).

With the links inside the Wallaby Tests output channel, custom links options are quite limited (here is the feature request in the VS Code repo), but as you can see on the GIF above, wallaby does provide stack links (when it can).

In your case it looks like the error stack is not displayed. It may or may not be specific to your tests/environment. If you run plain mocha and it does it display the error stacks correctly in the terminal, wallaby should be able to do it as well. If it's not the case, could you please share a sample project (attached or on GitHub) where I could reproduce it? Happy to have a look.

[Sequoia]

It looks like it is outputting stack traces but the async nature of the tests is causing the stack not to walk all the way back into the test file:

  1) Send Digest Emails getSubscribedClients gets only subscribed clients:
     MongoError: limit value must be non-negative
      at Function.MongoError.create (node_modules/mongodb-core/lib/error.js:31:11)
      at queryCallback (node_modules/mongodb-core/lib/cursor.js:281:36)
      at Callbacks.emit (node_modules/mongodb-core/lib/topologies/server.js:95:3)
      at null.messageHandler (node_modules/mongodb-core/lib/topologies/server.js:249:23)
      at Socket.<anonymous> (node_modules/mongodb-core/lib/connection/connection.js:265:22)
      at readableAddChunk (_stream_readable.js:153:18)
      at Socket.Readable.push (_stream_readable.js:111:10)
      at TCP.onread (net.js:531:20)
      at TCP.onread (node_modules/async-listener/glue.js:188:31)

[Sequoia]

Tests

describe.only('test with async', function(){
  it('prints stack with promise.reject', function(){
    return Promise.reject('bad!! :(');
  })
  it('prints stack with new promise reject with timeout', function(){
    return new Promise(function(resolve, reject){
      setTimeout(reject.bind(null, new Error('rejected local 1')),1000);
    });
  })
  it('prints stack with new promise reject in EXTERNAL file', function(){
    return require('./rejector.js')();
  })
  it('prints stack with new promise reject in EXTERNAL file accessing async API', function(){
    return require('./fs-rejector.js')();
  })
  it('prints stack with done(error)', function(done){
    setTimeout(function(){
      done(new Error('rejected local 2'));
    },1000);
  })
})

rejector.js

module.exports = function getRejector(){
  return new Promise(function(resolve, reject){
    setTimeout(reject.bind(null, new Error('rejected EXTERNAL')),1000);
  });
};

fs-rejector.js

const fs = require('fs');

module.exports = function getRejector(){
  return new Promise(function(resolve, reject){
    fs.readFile('not-exist',function(err,data){
      if(err){
        return reject(err);
      }
      resolve(data);
    });
  });
};

Output

  test with async
    1) prints stack with promise.reject
    2) prints stack with new promise reject with timeout
    3) prints stack with new promise reject in EXTERNAL file
    4) prints stack with new promise reject in EXTERNAL file accessing async API
    5) prints stack with done(error)


  0 passing (3s)
  5 failing

  1) test with async prints stack with promise.reject:
     Error: the string "bad!! :(" was thrown, throw an Error :)
      at propagateAslWrapper (node_modules/async-listener/index.js:394:23)
      at node_modules/async-listener/glue.js:188:31
      at node_modules/async-listener/index.js:431:70
      at process.fallback (node_modules/async-listener/index.js:450:15)

  2) test with async prints stack with new promise reject with timeout:
     Error: rejected local 1
      at test/agent.spec.js:20:36
      at new wrappedPromise (node_modules/async-listener/index.js:347:16)
      at Context.<anonymous> (test/agent.spec.js:19:12)

  3) test with async prints stack with new promise reject in EXTERNAL file:
     Error: rejected EXTERNAL
      at test/rejector.js:3:34
      at new wrappedPromise (node_modules/async-listener/index.js:347:16)
      at getRejector (test/rejector.js:2:10)
      at Context.<anonymous> (test/agent.spec.js:24:36)

  4) test with async prints stack with new promise reject in EXTERNAL file accessing async API:
     Error: ENOENT: no such file or directory, open 'not-exist'
      at Error (native)

  5) test with async prints stack with done(error):
     Error: rejected local 2
      at Timeout.<anonymous> (test/agent.spec.js:31:12)
      at Timeout._onTimeout (node_modules/async-listener/glue.js:188:31)

Prints stack back to /test.agent.spec.js?

1. ❌ prints stack with promise.reject
2. ✅ prints stack with new promise reject with timeout
3. ✅ prints stack with new promise reject in EXTERNAL file
4. ❌ prints stack with new promise reject in EXTERNAL file accessing async API
5. ✅ prints stack with done(error)

[Sequoia]

Not sure if this is a mocha bug or a node.js bug

[ArtemGovorov]

Looks like plain mocha is pretty consistent with wallaby:

(note that you're using VS Code 1.2 and the inline messages will only work starting from VS Code 1.3, you may download the insiders build of the editor if you like)

1. ❌ prints stack with promise.reject

This one is simple, you are rejecting with a string, so the testing framework can't get the stack trace. If you change it to reject with an error, it should be better (in both plain mocha and wallaby):

  it('prints stack with promise.reject', function () {
    return Promise.reject(new Error('bad!! :('));
  })

4. ❌ prints stack with new promise reject in EXTERNAL file accessing async API

This one is just a limitation of native ES6 promises. When the error is raised asynchronously, native promise just doesn't keep the previous stack. One way to fix it would be to use an enhancer of the native promises, such as bluebird. You don't have to use it prod (if you don't need nice stack traces there), and may only use it for testing, for example for wallaby, all you need to do is to install it and configure in the setup function:

module.exports = function () {
  return {
    ...
    setup: function(wallaby){
        global.Promise = require('bluebird');
        Promise.longStackTraces();
    }
  };
};

npm i bluebird --save-dev

Please let me know if addresses the raised issue and answers your question.

[Sequoia]

I think that answers my questions, thanks!