How to access the impact of a test failure?

[gsandie]

Sorry if this isn't the right place to ask.

I'm looking to implement inspec and been working with the cli to perform some initial testing. I can see that when I write tests with controls and impact the json summary (via inspec json . in my test directory) will display the rules with their impact. When I run the tests via inspect exec I don't see the impact of failing checks.

I've experimented with using the runner via ruby and using the json output formatter, however it doesn't show the test impact in it's output (only the control name as the ID which is handy).

Am I missing something? Or should I just be combining the json output together with the json summary to get a view of the impact of failing tests?

Thanks.

[chris-rock]

@gsandie Would love to hear more about the use case that you are trying to tackle.

Since we use RSpec under the hood as executor, we are somehow limited with the features that RSpec offers. We are thinking a lot about improvements. The current way would be to combine the json output with the test output.

[gsandie]

@chris-rock Thanks for the feedback.

I was looking at how we can integrate with our internal tools. So we could run the tests against the infrastructure and if there are any failures we can alert with the control and it's impact. We'd also have the information available on a dashboard.

[arothian]

To be honest, I expected some of the output to include the control information. I don't quite get the usefulness of the control related meta-data without it other than to point at the test suite to provide evidence that various controls are being checked.

I think it makes sense to have some sort of report that could be generated when running exec that ties back to the control metadata.

[chris-rock]

@arothian Yes, I agree, again this has to do with some internals of RSpec. We want to fix this, but it is not so easy to do. For now, I recommend to use Chef Compliance to generate a report.

[gsandie]

@chris-rock @arlimus - Thanks for 903b059 it helps me a lot 👍

[chris-rock]

@gsandie and @arothian we updated the json formatter in #671 It will be released with the next version of inspec. That should provide all the information necessary to parse it properly. Please reopen if we forgot something.