Handle failed assertions and include them in reports #86
Comments
|
Do you mind posting the result returned by phantomas ( can not dig into it right now )? @macbre has probably something to say about that, too? ;) |
|
Sure thing. I misspoke earlier and have edited my first post. It's actually the console.log of |
|
Here's a Gruntfile you can use to reproduce. Run I'm on grunt-phantomas module.exports = function(grunt) {
// Project configuration.
grunt.initConfig({
pkg: grunt.file.readJSON('package.json'),
phantomas: {
default: {
options: {
indexPath: './phantomas/',
options: {},
url: 'http://gruntjs.com/'
}
},
requests: {
options: {
indexPath: './phantomas/',
options: {
'assert-requests': 1
},
url: 'http://gruntjs.com/'
}
},
}
});
grunt.loadNpmTasks('grunt-phantomas');
grunt.registerTask('default', ['phantomas:default']);
}; |
|
Okay. May I ask you for an update on your side, also? When we're not on the same version, it will be hard to discuss later on... |
|
No problem, gee I didn't realize I was so far out of date. At a glance it looks pretty close to the same thing. Here's the output of |
|
@stefanjudis, correct me if I'm wrong, but the current version of phantomas-grunt does not use multiple runs feature from phantomas. Maybe it's worth porting phantomas-grunt to it (when all metrics stats are available on phantomas side). |
|
@macbre jap, you're right. I tried but am depending on this one macbre/phantomas#285. |
|
Depending on macbre/phantomas#314 and macbre/phantomas#315. |
|
Done :) |
|
Also of interest to this discussion: https://github.com/gmetais/grunt-devperf/ |
|
probably next week I've got time to tackle this issue. :) |
|
@rupl How to deal with failing assertion depending on data type ( median, max, min, ... ). Currently a warning is shown when one value out of 5 hit the border, but the graph will probably show a circle below the border, when in So I'm thinking to make the warnings on top only to show up when median value failed the assertion? What should be the value to show warnings? Max/Min seem to be useless to me, so I'd go for |
|
|
|
Wow, fantastic job on this thing! I agree with you that median is the best choice for determining pass/fail. That's that the graph plots, so it makes sense to me. |
|
Cool. I'm just doing some polishing and coverage increasements, but I think could be shipped monday or something. ;) |
|
What do you think of that screenshot (all colors have to be adjusted)?
|
|
I think having a toggle to do that would be great. Especially useful if someone displayed the reports on a TV or other big monitor somewhere. But yeah, it might be too much for someone sitting in front of their computer. |
|
Or maybe more something like this?
|
Let's make a follow up then. ;) |
|
ah schön! the outline a nice balance while still making the failures noticable |
|
Here we are. ;) http://stefanjudis.github.io/grunt-phantomas/gruntjs/ Release planned definitely monday/tuesday. Now it's |
|
You are amazing. Enjoy |
|
Great work.. ! Two things came to my mind while looking at it.. First it would be really nice to be able to initially hide all the tables. At least i want to quick scan all the graphs to get an overview and investigate further (using the tables) if something shows up. Which leads me to the second thing.. It think it would be really helpful to show a scale on the left/right side of a graph AND scale it up to the largest value.
is really hard to interpret without hovering a dot to see a tooltip. best regards |
|
I think those are good ideas, would you mind filing a separate issue? Individual scaling for graphs was in the back of my head too. |
|
Thanks guys for feedback. Jap, let's please make small issues and discuss if needed there. :) |
…ling added assertion handling - fixes #86
|
@rupl @davidlinse merged and released. |
|
|
|
And by the way guys, I'm planning to marry phantomas with psi in a seperate project without grunt dependency. |
|
oh man that would be awesome, looking forward to seeing it! |
|
Come on. Say "I'm in." :) But no pressure. |
|
I'm in |
|
Nice... I'm planning to kick it off in 2-4 weeks. You'll receive an collaboration invite in a sec. :) Name is also not fixed, yet. But I really like that name. |
I think it would be really great to handle failed assertions and log them the same way a successful run is logged. We could even color-code the data points that fail assertions.
This is the output I see when I set an assertion that fails. In this case I set
assert-requeststo a very low number in theoptionsobject of the Gruntfile:I started hunting around for a solution and noticed that
metricsdoesn't exist when the assertion fails, which is theundefinedreported in the Grunt output. Instead thepromiseresultsargument contains a small-ish object with only a couple properties: the Error code (mine was Error 1), and a stack trace.The text was updated successfully, but these errors were encountered: