Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.
Sign upHandle failed assertions and include them in reports #86
Comments
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
stefanjudis
May 15, 2014
Owner
Do you mind posting the result returned by phantomas ( can not dig into it right now )?
How it is now and how it could be, to work with it... :)
@macbre has probably something to say about that, too? ;)
|
Do you mind posting the result returned by phantomas ( can not dig into it right now )? @macbre has probably something to say about that, too? ;) |
stefanjudis
added
the
enhancement
label
May 15, 2014
stefanjudis
self-assigned this
May 15, 2014
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
rupl
May 15, 2014
Contributor
Sure thing. I misspoke earlier and have edited my first post. It's actually the console.log of results before return new Promise..., not the metrics object of the fulfilledPromise. results is an array of objects corresponding to the number of blended runs. Since they're all identical, here is one of the objects inside results:
{ _bitField: 134217728,
_settledValue:
{ name: 'RejectionError',
message: '1',
cause: [Error: 1],
stack: 'Error: 1\n at maybeWrapAsError (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/util.js:128:12)\n at PromiseResolver$_callback (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/promise_resolver.js:55:48)\n at ChildProcess.<anonymous> (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/phantomas/lib/index.js:118:4)\n at ChildProcess.EventEmitter.emit (events.js:98:17)\n at maybeClose (child_process.js:743:16)\n at Socket.<anonymous> (child_process.js:956:11)\n at Socket.EventEmitter.emit (events.js:95:17)\n at Pipe.close (net.js:465:12)' } }
|
Sure thing. I misspoke earlier and have edited my first post. It's actually the console.log of |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
rupl
May 15, 2014
Contributor
Here's a Gruntfile you can use to reproduce. Run grunt phantomas:requests
I'm on grunt-phantomas 0.5.3, the copy of Phantomas inside the node_modules dir says version 0.12.1
module.exports = function(grunt) {
// Project configuration.
grunt.initConfig({
pkg: grunt.file.readJSON('package.json'),
phantomas: {
default: {
options: {
indexPath: './phantomas/',
options: {},
url: 'http://gruntjs.com/'
}
},
requests: {
options: {
indexPath: './phantomas/',
options: {
'assert-requests': 1
},
url: 'http://gruntjs.com/'
}
},
}
});
grunt.loadNpmTasks('grunt-phantomas');
grunt.registerTask('default', ['phantomas:default']);
};|
Here's a Gruntfile you can use to reproduce. Run I'm on grunt-phantomas module.exports = function(grunt) {
// Project configuration.
grunt.initConfig({
pkg: grunt.file.readJSON('package.json'),
phantomas: {
default: {
options: {
indexPath: './phantomas/',
options: {},
url: 'http://gruntjs.com/'
}
},
requests: {
options: {
indexPath: './phantomas/',
options: {
'assert-requests': 1
},
url: 'http://gruntjs.com/'
}
},
}
});
grunt.loadNpmTasks('grunt-phantomas');
grunt.registerTask('default', ['phantomas:default']);
}; |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
stefanjudis
May 15, 2014
Owner
Okay. May I ask you for an update on your side, also?
When we're not on the same version, it will be hard to discuss later on...
But that's already enough for me to dig into it. :)
|
Okay. May I ask you for an update on your side, also? When we're not on the same version, it will be hard to discuss later on... |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
rupl
May 15, 2014
Contributor
No problem, gee I didn't realize I was so far out of date. At a glance it looks pretty close to the same thing. Here's the output of 0.7.0 with console.log(requests) added back in:
$ grunt phantomas:requests
Running "phantomas:requests" (phantomas) task
PHANTOMAS EXECUTION(S) STARTED.
>> 5 Phantomas execution(s) done -> checking results:
[ { _bitField: 134217728,
_settledValue:
{ name: 'RejectionError',
message: '1',
cause: [Error: 1],
stack: 'Error: 1\n at maybeWrapAsError (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/util.js:127:12)\n at PromiseResolver$_callback (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/promise_resolver.js:55:48)\n at ChildProcess.<anonymous> (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/phantomas/lib/index.js:165:4)\n at ChildProcess.EventEmitter.emit (events.js:98:17)\n at maybeClose (child_process.js:743:16)\n at Socket.<anonymous> (child_process.js:956:11)\n at Socket.EventEmitter.emit (events.js:95:17)\n at Pipe.close (net.js:465:12)' } },
{ _bitField: 134217728,
_settledValue:
{ name: 'RejectionError',
message: '1',
cause: [Error: 1],
stack: 'Error: 1\n at maybeWrapAsError (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/util.js:127:12)\n at PromiseResolver$_callback (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/promise_resolver.js:55:48)\n at ChildProcess.<anonymous> (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/phantomas/lib/index.js:165:4)\n at ChildProcess.EventEmitter.emit (events.js:98:17)\n at maybeClose (child_process.js:743:16)\n at Process.ChildProcess._handle.onexit (child_process.js:810:5)' } },
{ _bitField: 134217728,
_settledValue:
{ name: 'RejectionError',
message: '1',
cause: [Error: 1],
stack: 'Error: 1\n at maybeWrapAsError (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/util.js:127:12)\n at PromiseResolver$_callback (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/promise_resolver.js:55:48)\n at ChildProcess.<anonymous> (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/phantomas/lib/index.js:165:4)\n at ChildProcess.EventEmitter.emit (events.js:98:17)\n at maybeClose (child_process.js:743:16)\n at Socket.<anonymous> (child_process.js:956:11)\n at Socket.EventEmitter.emit (events.js:95:17)\n at Pipe.close (net.js:465:12)' } },
{ _bitField: 134217728,
_settledValue:
{ name: 'RejectionError',
message: '1',
cause: [Error: 1],
stack: 'Error: 1\n at maybeWrapAsError (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/util.js:127:12)\n at PromiseResolver$_callback (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/promise_resolver.js:55:48)\n at ChildProcess.<anonymous> (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/phantomas/lib/index.js:165:4)\n at ChildProcess.EventEmitter.emit (events.js:98:17)\n at maybeClose (child_process.js:743:16)\n at Socket.<anonymous> (child_process.js:956:11)\n at Socket.EventEmitter.emit (events.js:95:17)\n at Pipe.close (net.js:465:12)' } },
{ _bitField: 134217728,
_settledValue:
{ name: 'RejectionError',
message: '1',
cause: [Error: 1],
stack: 'Error: 1\n at maybeWrapAsError (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/util.js:127:12)\n at PromiseResolver$_callback (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/bluebird/js/main/promise_resolver.js:55:48)\n at ChildProcess.<anonymous> (.../grunt/phantomas/node_modules/grunt-phantomas/node_modules/phantomas/lib/index.js:165:4)\n at ChildProcess.EventEmitter.emit (events.js:98:17)\n at maybeClose (child_process.js:743:16)\n at Socket.<anonymous> (child_process.js:956:11)\n at Socket.EventEmitter.emit (events.js:95:17)\n at Pipe.close (net.js:465:12)' } } ]
>> Phantomas execution not successful -> RejectionError: 1
>> Phantomas execution not successful -> RejectionError: 1
>> Phantomas execution not successful -> RejectionError: 1
>> Phantomas execution not successful -> RejectionError: 1
>> Phantomas execution not successful -> RejectionError: 1
>> SOMETHING WENT WRONG...
>> No run was successful.
|
No problem, gee I didn't realize I was so far out of date. At a glance it looks pretty close to the same thing. Here's the output of |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
macbre
May 15, 2014
@stefanjudis, correct me if I'm wrong, but the current version of phantomas-grunt does not use multiple runs feature from phantomas. Maybe it's worth porting phantomas-grunt to it (when all metrics stats are available on phantomas side).
macbre
commented
May 15, 2014
|
@stefanjudis, correct me if I'm wrong, but the current version of phantomas-grunt does not use multiple runs feature from phantomas. Maybe it's worth porting phantomas-grunt to it (when all metrics stats are available on phantomas side). |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
stefanjudis
May 15, 2014
Owner
@macbre jap, you're right.
I tried but am depending on this one macbre/phantomas#285.
As far as I remember. :)
|
@macbre jap, you're right. I tried but am depending on this one macbre/phantomas#285. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
|
Depending on macbre/phantomas#314 and macbre/phantomas#315. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
macbre
commented
May 17, 2014
|
Done :) |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
rupl
May 28, 2014
Contributor
Also of interest to this discussion: https://github.com/gmetais/grunt-devperf/
|
Also of interest to this discussion: https://github.com/gmetais/grunt-devperf/ |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
|
probably next week I've got time to tackle this issue. :) |
stefanjudis
added this to the v0.8.0 milestone
Jun 28, 2014
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
stefanjudis
Jun 28, 2014
Owner
@rupl
Okay. I'm almost there. One question popped up.
How to deal with failing assertion depending on data type ( median, max, min, ... ).
Currently a warning is shown when one value out of 5 hit the border, but the graph will probably show a circle below the border, when in median mode.
So I'm thinking to make the warnings on top only to show up when median value failed the assertion?
Additionally it's the initial state, so in my mind it makes sense.
What should be the value to show warnings? Max/Min seem to be useless to me, so I'd go for median? what do you think?
|
@rupl How to deal with failing assertion depending on data type ( median, max, min, ... ). Currently a warning is shown when one value out of 5 hit the border, but the graph will probably show a circle below the border, when in So I'm thinking to make the warnings on top only to show up when median value failed the assertion? What should be the value to show warnings? Max/Min seem to be useless to me, so I'd go for |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
|
|
added a commit
that referenced
this issue
Jun 28, 2014
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
rupl
Jun 28, 2014
Contributor
Wow, fantastic job on this thing!
I agree with you that median is the best choice for determining pass/fail. That's that the graph plots, so it makes sense to me.
|
Wow, fantastic job on this thing! I agree with you that median is the best choice for determining pass/fail. That's that the graph plots, so it makes sense to me. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
stefanjudis
Jun 28, 2014
Owner
Cool. I'm just doing some polishing and coverage increasements, but I think could be shipped monday or something. ;)
|
Cool. I'm just doing some polishing and coverage increasements, but I think could be shipped monday or something. ;) |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
stefanjudis
Jun 28, 2014
Owner
What do you think of that screenshot (all colors have to be adjusted)?
I'm playing around with really marking failed graphs, but I'm not sure if it's too much?
|
What do you think of that screenshot (all colors have to be adjusted)?
|
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
rupl
Jun 28, 2014
Contributor
I think having a toggle to do that would be great. Especially useful if someone displayed the reports on a TV or other big monitor somewhere. But yeah, it might be too much for someone sitting in front of their computer.
|
I think having a toggle to do that would be great. Especially useful if someone displayed the reports on a TV or other big monitor somewhere. But yeah, it might be too much for someone sitting in front of their computer. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
|
Or maybe more something like this?
|
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
stefanjudis
Jun 28, 2014
Owner
I think having a toggle to do that would be great.
Let's make a follow up then. ;)
Let's make a follow up then. ;) |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
rupl
Jun 28, 2014
Contributor
ah schön! the outline a nice balance while still making the failures noticable
|
ah schön! the outline a nice balance while still making the failures noticable |
added a commit
that referenced
this issue
Jun 28, 2014
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
stefanjudis
Jun 28, 2014
Owner
Here we are. ;)
http://stefanjudis.github.io/grunt-phantomas/gruntjs/
Release planned definitely monday/tuesday. Now it's
|
Here we are. ;) http://stefanjudis.github.io/grunt-phantomas/gruntjs/ Release planned definitely monday/tuesday. Now it's |
added a commit
that referenced
this issue
Jun 28, 2014
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
|
You are amazing. Enjoy |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
davidlinse
Jun 28, 2014
Great work.. !
Two things came to my mind while looking at it..
First it would be really nice to be able to initially hide all the tables. At least i want to quick scan all the graphs to get an overview and investigate further (using the tables) if something shows up.
Which leads me to the second thing.. It think it would be really helpful to show a scale on the left/right side of a graph AND scale it up to the largest value.
A graph like:
is really hard to interpret without hovering a dot to see a tooltip.
What do you think ?
best regards
~david
davidlinse
commented
Jun 28, 2014
|
Great work.. ! Two things came to my mind while looking at it.. First it would be really nice to be able to initially hide all the tables. At least i want to quick scan all the graphs to get an overview and investigate further (using the tables) if something shows up. Which leads me to the second thing.. It think it would be really helpful to show a scale on the left/right side of a graph AND scale it up to the largest value.
is really hard to interpret without hovering a dot to see a tooltip. best regards |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
rupl
Jun 28, 2014
Contributor
I think those are good ideas, would you mind filing a separate issue? Individual scaling for graphs was in the back of my head too.
|
I think those are good ideas, would you mind filing a separate issue? Individual scaling for graphs was in the back of my head too. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
stefanjudis
Jun 28, 2014
Owner
Thanks guys for feedback.
Jap, let's please make small issues and discuss if needed there. :)
|
Thanks guys for feedback. Jap, let's please make small issues and discuss if needed there. :) |
added a commit
that referenced
this issue
Jun 29, 2014
added a commit
that referenced
this issue
Jun 29, 2014
added a commit
that referenced
this issue
Jun 29, 2014
added a commit
that referenced
this issue
Jun 30, 2014
stefanjudis
closed this
in
dd9320c
Jun 30, 2014
added a commit
that referenced
this issue
Jun 30, 2014
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
|
@rupl @davidlinse merged and released. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
davidlinse
commented
Jun 30, 2014
|
|
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
stefanjudis
Jul 1, 2014
Owner
And by the way guys,
I'm planning to marry phantomas with psi in a seperate project without grunt dependency.
And collaborators would be awesome... :)
|
And by the way guys, I'm planning to marry phantomas with psi in a seperate project without grunt dependency. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
|
oh man that would be awesome, looking forward to seeing it! |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
|
Come on. Say "I'm in." :) But no pressure. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
|
I'm in |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
stefanjudis
Jul 1, 2014
Owner
Nice... I'm planning to kick it off in 2-4 weeks.
And I'm fine by removing grunt by myself. After that I'd need some supports with tickets, ideas and interface.
You'll receive an collaboration invite in a sec. :)
Name is also not fixed, yet. But I really like that name.
|
Nice... I'm planning to kick it off in 2-4 weeks. You'll receive an collaboration invite in a sec. :) Name is also not fixed, yet. But I really like that name. |
rupl commentedMay 15, 2014
I think it would be really great to handle failed assertions and log them the same way a successful run is logged. We could even color-code the data points that fail assertions.
This is the output I see when I set an assertion that fails. In this case I set
assert-requeststo a very low number in theoptionsobject of the Gruntfile:I started hunting around for a solution and noticed that
metricsdoesn't exist when the assertion fails, which is theundefinedreported in the Grunt output. Instead thepromiseresultsargument contains a small-ish object with only a couple properties: the Error code (mine was Error 1), and a stack trace.