Amended Publish Test Results task to fail build if test failures are present in the attached files.

[T-rav]

This pull request is to address issue #1268
The recommended solutions did not fix my version of this issue.
I have an external process, Katalon Studio, that runs UI test and does not report back the test run status to TFS.
This means I need to rely upon the Publish Test Results task being more intelligent and failing the build if test failures are detected in the attachments.

I did look at other proposed solutions in the issue and they are inadequate or not easy to repeat across builds.

Taking @alejandro5042 suggetion I added a check box.

If FAILED is detected, in any of the attachments, the build is marked as failed.

[msftclas]

All CLA requirements met.

[MirzaSikander]

This is great! When will this be available. I currently don't see it in the Publish Test Results Task.

[prawalagarwal]

@T-rav I am afraid this is not the right fix. Based on the runner you can have different content in the result files. For eg. for VsTest runner, trx files have aggregated reporting in the end which says "Failed=0" when there are no failed tests. With your fix, this will too fail the task. Also, this match is case sensitive which is again based on the runner. @vinodjo will update the issues thread with the latest information on this.

[krishnand]

@vinodjo - Do we know when this is going to be rolled out?.

[vinodjo]

@krishnand, The purpose of Publish Test Results Task is to report the test results, hence if task is able to report all the results successfully it is marked as success even if it there were test failures. Since we have received a few requests on this, I have added this to our backlog. However this is more than just looking for failed test, but also multiple result formats along with multiple outcomes that could be treated as failures (other than failed). Unfortunately at this stage I don’t have any concrete information on the timeline to share with you.

In the interim, here is a sample that queries the failed tests and returns failure, which you can refer to enable the expected behavior

[lqueryvg]

Wow. Why would anyone not want this feature ?
I'm struggling to get my head around this...
Again... why would anyone want a build definition which runs tests and always succeeds, even if the tests failed ???

You could at least implement an option (as per this pull request).
Or create a variable which a subsequent task can read and make use of.

If you (Microsoft) are saying that the purpose of the publish task is ONLY to publish results, then where is the task which helps the user interpret the test results ? Or is every user expected to parse the test output in their own way ?

Surely this publish task is the obvious and natural place to implement this functionality ?

[stephenmichaelf]

@prawalagarwal Should we close this?

[krlydm]

Any update on this?

[stephenmichaelf]

Closing due to age, please reopen if needed.