Feature request '--failure-report-only'

[digitaldonkey]

With out pipeline we regularly need to download failed tests with 600MB+, while only requiring a single failed screen to update.

@garris Can you imagine a change like an extra param (e.g. --failure-report-only) where only the failed tests will be aggregated in a test report?

That would be kind of eco in many ways.

[garris]

Just so I understand the question -- are you talking about downloading the html report directory with all the images etc?

[digitaldonkey]

Yes. A reduced report only containing failures. Garris <notifications@github.com> schrieb am Fr., 22. Jan. 2021, 17:15:

…

Just so I understand the question -- are you talking about downloading the html report directory with all the images etc? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1272 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADMSLXOWO3PMYZOPEDZUHDS3GQDTANCNFSM4WOQENLA> .

[garris]

Hmm -- what is the carbon footprint of moving 600M of useless screenshots across the internet multiple times a day everyday? 😅

Anyway -- yes, this is a totally valid request and there should be a solution for this. I'd suggest that it is not added to backstop executable per se. I think it could very easily be implemented as a node script which rewrites the report JSON and prunes the images directory before you run your copy command.

I can imagine a straight forward implementation... check out the screenshot below -- I am showing two similar files.

• The one on the left is a JSONP file used by your browser when displaying a report.
• The one on the right is a plain JSON version of the above file (it's there for a situation just like this)

The script could...

1. Make a local copy of backstop_data (excluding bitmap_reference, bitmap_test, engine_scripts). Make the copy the working directory. Then...
2. Using the file on the right, copy only the images you need into your new backstop_data copy using source paths in pair.test and pair.reference where pair.status === "fail". Then...
3. Using the file on the right, regenerate the file on the left filtering out any results where pair.status === "pass"

Something like that.

If you build this I am sure other users would be interested to see how you did it. Would be great to explain how to use it in the readme etc.

Please let me know if this helps!

[digitaldonkey]

Thank you. I will try it.. but it may take some time.

[fuhlig]

I'd suggest that it is not added to backstop executable per se. I think it could very easily be implemented as a node script which rewrites the report JSON and prunes the images directory before you run your copy command.

Like stated above, this is a very valid and valuable request.
It being integrated in backstop would remove the need for user-implemented solutions and extra hoops needed for first including & then exluding passed scenarios to the report.

I'd be happy to collaborate on this feature.

[garris]

@fuhlig Where do you see the entry point for this as a built-in feature? How would you manage the extra/alternate assets?

[fuhlig]

Just brainstorming here, without knowing the current internals...
If the option is passed in, only the scalenarios containing a diff get picked up by the report. Passed scenarios are not included and could be even removed after comparison.

[fuhlig]

@fuhlig Where do you see the entry point for this as a built-in feature? How would you manage the extra/alternate assets?

What do you refer to by extra/alternate assets?

[garris]

Extra/alternate assets are files and images. I think what @digitaldonkey is asking for is a portable, encapsulated report that is pruned down to only include failing test data -- optimized for easy copying. The solution I proposed is to create a new directory (using filtered copy operations and rewriting a jsonp file). The reason I suggested this approach is because one would not have to know the internals to implement this feature -- this approach could probably be done in say, 2 hours. The road you are going down would require you to understand how the internals work and that would be a boring and laborious task for you that would take at least 2 hours in itself. Then, in the end, after looking at all that internal nonsense you might simply choose to do what I suggested anyway. This is not a high priority but would be happy if you want to contribute.

[digitaldonkey]

Did something here https://github.com/digitaldonkey/backstopjs-failsonly
It's not yet really tested so consider it pre-alpha. Colaborators welcome.

[garris]

@digitaldonkey i will look at Thai tomorrow.

[garris]

@digitaldonkey Reviewed the code today -- looks pretty good. Will try to install this weekend -- if it works I'm happy to promote it in the backstop docs. Cheers.