Handle failed assertions and include them in reports

[rupl]

I think it would be really great to handle failed assertions and log them the same way a successful run is logged. We could even color-code the data points that fail assertions.

This is the output I see when I set an assertion that fails. In this case I set assert-requests to a very low number in the options object of the Gruntfile:

$ grunt phantomas:requests
Running "phantomas:requests" (phantomas) task

PHANTOMAS EXECUTION(S) STARTED.
>> 5 Phantomas execution(s) done -> checking results:
>> SOMETHING WENT WRONG...
>> TypeError: Cannot call method 'value' of undefined
>> TypeError: Cannot call method 'value' of undefined
>>     at null.<anonymous> (.../grunt/phantomas/node_modules/grunt-phantomas/tasks/lib/phantomas.js:377:50)
>>     at tryCatch2 (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/util.js:74:19)
>>     at Promise$_resolveFromResolver [as _resolveFromResolver] (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/promise.js:608:13)
>>     at new Promise (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/promise.js:88:37)
>>     at Phantomas.formResult (.../grunt/phantomas/node_modules/grunt-phantomas/tasks/lib/phantomas.js:372:10)
>>     at tryCatch1 (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/util.js:64:19)
>>     at Promise$_callHandler [as _callHandler] (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/promise.js:708:13)
>>     at Promise$_settlePromiseFromHandler [as _settlePromiseFromHandler] (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/promise.js:724:18)
>>     at Promise$_settlePromiseAt [as _settlePromiseAt] (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/promise.js:896:14)
>>     at Promise$_fulfillPromises [as _fulfillPromises] (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/promise.js:1041:14)

I started hunting around for a solution and noticed that metrics doesn't exist when the assertion fails, which is the undefined reported in the Grunt output. Instead the promise results argument contains a small-ish object with only a couple properties: the Error code (mine was Error 1), and a stack trace.

[stefanjudis]

Do you mind posting the result returned by phantomas ( can not dig into it right now )?
How it is now and how it could be, to work with it... :)

@macbre has probably something to say about that, too? ;)

[rupl]

Sure thing. I misspoke earlier and have edited my first post. It's actually the console.log of results before return new Promise..., not the metrics object of the fulfilledPromise. results is an array of objects corresponding to the number of blended runs. Since they're all identical, here is one of the objects inside results:

{ _bitField: 134217728,
    _settledValue:
     { name: 'RejectionError',
       message: '1',
       cause: [Error: 1],
       stack: 'Error: 1\n    at maybeWrapAsError (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/util.js:128:12)\n    at PromiseResolver$_callback (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/promise_resolver.js:55:48)\n    at ChildProcess.<anonymous> (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/phantomas/lib/index.js:118:4)\n    at ChildProcess.EventEmitter.emit (events.js:98:17)\n    at maybeClose (child_process.js:743:16)\n    at Socket.<anonymous> (child_process.js:956:11)\n    at Socket.EventEmitter.emit (events.js:95:17)\n    at Pipe.close (net.js:465:12)' } }

[rupl]

Here's a Gruntfile you can use to reproduce. Run grunt phantomas:requests

I'm on grunt-phantomas 0.5.3, the copy of Phantomas inside the node_modules dir says version 0.12.1

module.exports = function(grunt) {

  // Project configuration.
  grunt.initConfig({
    pkg: grunt.file.readJSON('package.json'),

    phantomas: {
      default: {
        options: {
          indexPath: './phantomas/',
          options: {},
          url: 'http://gruntjs.com/'
        }
      },
      requests: {
        options: {
          indexPath: './phantomas/',
          options: {
            'assert-requests': 1
          },
          url: 'http://gruntjs.com/'
        }
      },
    }
  });

  grunt.loadNpmTasks('grunt-phantomas');
  grunt.registerTask('default', ['phantomas:default']);
};

[stefanjudis]

Okay. May I ask you for an update on your side, also?

When we're not on the same version, it will be hard to discuss later on...
But that's already enough for me to dig into it. :)

[rupl]

No problem, gee I didn't realize I was so far out of date. At a glance it looks pretty close to the same thing. Here's the output of 0.7.0 with console.log(requests) added back in:

$ grunt phantomas:requests
Running "phantomas:requests" (phantomas) task

PHANTOMAS EXECUTION(S) STARTED.
>> 5 Phantomas execution(s) done -> checking results:
[ { _bitField: 134217728,
    _settledValue:
     { name: 'RejectionError',
       message: '1',
       cause: [Error: 1],
       stack: 'Error: 1\n    at maybeWrapAsError (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/util.js:127:12)\n    at PromiseResolver$_callback (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/promise_resolver.js:55:48)\n    at ChildProcess.<anonymous> (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/phantomas/lib/index.js:165:4)\n    at ChildProcess.EventEmitter.emit (events.js:98:17)\n    at maybeClose (child_process.js:743:16)\n    at Socket.<anonymous> (child_process.js:956:11)\n    at Socket.EventEmitter.emit (events.js:95:17)\n    at Pipe.close (net.js:465:12)' } },
  { _bitField: 134217728,
    _settledValue:
     { name: 'RejectionError',
       message: '1',
       cause: [Error: 1],
       stack: 'Error: 1\n    at maybeWrapAsError (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/util.js:127:12)\n    at PromiseResolver$_callback (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/promise_resolver.js:55:48)\n    at ChildProcess.<anonymous> (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/phantomas/lib/index.js:165:4)\n    at ChildProcess.EventEmitter.emit (events.js:98:17)\n    at maybeClose (child_process.js:743:16)\n    at Process.ChildProcess._handle.onexit (child_process.js:810:5)' } },
  { _bitField: 134217728,
    _settledValue:
     { name: 'RejectionError',
       message: '1',
       cause: [Error: 1],
       stack: 'Error: 1\n    at maybeWrapAsError (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/util.js:127:12)\n    at PromiseResolver$_callback (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/promise_resolver.js:55:48)\n    at ChildProcess.<anonymous> (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/phantomas/lib/index.js:165:4)\n    at ChildProcess.EventEmitter.emit (events.js:98:17)\n    at maybeClose (child_process.js:743:16)\n    at Socket.<anonymous> (child_process.js:956:11)\n    at Socket.EventEmitter.emit (events.js:95:17)\n    at Pipe.close (net.js:465:12)' } },
  { _bitField: 134217728,
    _settledValue:
     { name: 'RejectionError',
       message: '1',
       cause: [Error: 1],
       stack: 'Error: 1\n    at maybeWrapAsError (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/util.js:127:12)\n    at PromiseResolver$_callback (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/promise_resolver.js:55:48)\n    at ChildProcess.<anonymous> (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/phantomas/lib/index.js:165:4)\n    at ChildProcess.EventEmitter.emit (events.js:98:17)\n    at maybeClose (child_process.js:743:16)\n    at Socket.<anonymous> (child_process.js:956:11)\n    at Socket.EventEmitter.emit (events.js:95:17)\n    at Pipe.close (net.js:465:12)' } },
  { _bitField: 134217728,
    _settledValue:
     { name: 'RejectionError',
       message: '1',
       cause: [Error: 1],
       stack: 'Error: 1\n    at maybeWrapAsError (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/util.js:127:12)\n    at PromiseResolver$_callback (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/promise_resolver.js:55:48)\n    at ChildProcess.<anonymous> (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/phantomas/lib/index.js:165:4)\n    at ChildProcess.EventEmitter.emit (events.js:98:17)\n    at maybeClose (child_process.js:743:16)\n    at Socket.<anonymous> (child_process.js:956:11)\n    at Socket.EventEmitter.emit (events.js:95:17)\n    at Pipe.close (net.js:465:12)' } } ]
>> Phantomas execution not successful -> RejectionError: 1
>> Phantomas execution not successful -> RejectionError: 1
>> Phantomas execution not successful -> RejectionError: 1
>> Phantomas execution not successful -> RejectionError: 1
>> Phantomas execution not successful -> RejectionError: 1
>> SOMETHING WENT WRONG...
>> No run was successful.

[macbre]

@stefanjudis, correct me if I'm wrong, but the current version of phantomas-grunt does not use multiple runs feature from phantomas. Maybe it's worth porting phantomas-grunt to it (when all metrics stats are available on phantomas side).

[stefanjudis]

@macbre jap, you're right.

I tried but am depending on this one macbre/phantomas#285.
As far as I remember. :)

[stefanjudis]

Depending on macbre/phantomas#314 and macbre/phantomas#315.

[macbre]

Done :)

[rupl]

Also of interest to this discussion: https://github.com/gmetais/grunt-devperf/

[stefanjudis]

👍

probably next week I've got time to tackle this issue. :)

[stefanjudis]

@rupl
Okay. I'm almost there. One question popped up.

How to deal with failing assertion depending on data type ( median, max, min, ... ).

Currently a warning is shown when one value out of 5 hit the border, but the graph will probably show a circle below the border, when in median mode.

So I'm thinking to make the warnings on top only to show up when median value failed the assertion?
Additionally it's the initial state, so in my mind it makes sense.

What should be the value to show warnings? Max/Min seem to be useless to me, so I'd go for median? what do you think?