Publishing test results with failed tests does not fail the build

[strtdusty]

I am using xunit test through DNX. For each project we generate a test result xml file and use the "publish test results" task to publish the results to the build. The results get published correctly and we can see that some tests failed. When tests fail, however, the build is not marked as failed. I could fail the build when running xunit, but the results would not get published. Is there a way for the build to look at the published test results and fail if there are test failures?

[chrispat]

You can mark the Publish Test Results task as Always Run and it will run even for a failed build.

[richardszalay]

@chrisrpatterson Your response is unrelated to @strtdusty's issue. The issue is that if a test fail, the "build" is still marked as successful. The successful build contains a failed test, sure, but it will go unnoticed so it's almost irrelevant.

[liamnichols]

+1 for this. I want to use the VSTS build system for our mobile projects but back it via Jenkins jobs as the Xcode support is somewhat lacking for my needs.

My problem is that I have to mark the Jenkins job as unstable as I can only download artefacts (my junit results) from the "last successful build". This then means that my build appears to have succeeded if the user doesn't actually click into the build to investigate.

[Yaash19]

@strtdusty

how did you resolve this ? i am having similar issue.

[strtdusty]

I used @chrisrpatterson 's suggestion. Our builds end up being marked as failed though which is correct.

[asinitson]

Why this is marked as closed? Although problem is resolved for the original request, the issue Publishing test results with failed tests does not fail the build is still not resolved as @liamnichols mentioned. Here are the steps to reproduce:

1. Create build definition with only one step: "Publish Test Results"
2. Run the build

Expected result:
If some of the tests are failed, then build should also fail.

Actual result:
Build always passes, irrespective of the results of the test.

Solution proposed above only works if you run the tests as a previous build step. But it is still impossible to fail the build purely depending on the published results.

Why is this important? If you have your build running outside of TFS bubble and just "publish" the results of the tests, then you still want to know a status of the build.

[pakdev]

@chrisrpatterson Can you please re-open this issue given @asinitson's rationale?

[alejandro5042]

There should at least be a checkbox for selecting whether the published test results can fail the build.

[kngrace]

I'm running two different test suites and I am able to fail the build with the test task itself for only one of them but not for the other. I'd like to be able to rely on the Publish Test Results task to mark a build failure if any of the tests it has published have failed.

[pakdev]

This is also a problem for Javascript tests/linting. My jshint/jscs failures are reported in the Tests tab, but the "npm test" and "Publish Test Results" tasks pass.

[asinitson]

Hey, everyone!

Since original bug reporter has his problem solved and this bug report is closed, I guess it makes sense to add another bug report. @chrisrpatterson: Is anybody looking at the comments for this closed bug report? Are you considering re-opening it?

I will wait a day or two for the reply and then create another report, so we at least would have some kind of statement: "won't fix" or "planned in some future release".

[ericsciple]

User voice is probably a better place for this feedback.

I'll re-open and assign to the appropriate team to see the feedback. This repo isn't used to track feature requests, so I would expect the issue to get closed again.

[alejandro5042]

@ericsciple I think this is a bug, not a feature request. Do you track bugs in this repo?

At any rate, I suggest making it clear in your README that the UserVoice (link?) is the primary place for feature feedback. I certainly didn't know that--others won't either. You could even consider disabling the issues tab if truly no one is checking these. Otherwise, it might be good to do at least a weekly review or risk appearing like Microsoft doesn't care about this repo. Just a friendly suggestion :)

[ncunning]

Is the solution to this from @chrisrpatterson not correct.
In my case: I use Command Line Step and Set the Task to "Fail on Standard Error".
Then ensure that the Publish Test Results Under Control Options -> "Run this Task" is set to "Even if a previous task has failed, unless the build was canceled"

I have this solution working for karma tests which fail yet still return the test results.
Using JUnit as my test reporter for an Angular Project.

Note: Also if running tests where don;t force a clean checkout. Ensure that there is a task to remove the test and coverage folders prior to running the tests otherwise the Publish task step will pick up the xml test results from a previous test

[pakdev]

@ncunning Interesting, I didn't know the Command Line task allowed "Fail on Standard Error". My workaround was to use/create "reporters" specific to VSTS (jscs-reporter-vso, jshint-vso-reporter) and to manually parse C# TRX files. Now, if a test failure is detected, logging commands are written to stdout to fail the build.

I think using the logging commands/Command Line task (as suggested by @ncunning) is the solution to this issue. It's more clear to users that their tests failed when the testing step fails vs. the publishing step. The only downside is that the test results need to be parsed twice: once to determine if there are failing tests and once in Publish Test Results.

[ncunning]

@pakdev When i look at the output it is the actual testing step which fails. The Publishing Tests Results allow the Tests section to display the charts and failed test etc..

[pakdev]

@ncunning Sorry, I meant our solutions are better than failing the Publish Test Results task as suggested by this issue. I've edited my response to reflect this.

[vinodjo]

The purpose of Publish Test Results Task is to report the test results, hence if task is able to report all the results successfully it is marked as success even if it there were test failures. If the test execution does not return failure for a failed test, here is a sample that queries the failed tests and returns failure, which you can refer to drive the expected behavior https://github.com/ManojBableshwar/VstsTestRestApiSamples/blob/master/QueryTestRunResultsAndFailBuild/QueryTestRunResultsAndFailBuild.cs

[vinodjo]

As suggested above, closing the issue.

[Nekobravo]

help´

[acesiddhu]

@Nekobravo please create a new issue

[DalSoft]

The new .NET Core VSTS tasks for dotnet test allow you to publish tests as part of the one step.

If you can't use them go with @chrisrpatterson suggestion Publish Test Results task as Always Run, then add the following command line task to fail the build:

if not "%agent.jobstatus%"=="Succeeded" (
    exit /b 666
)

[ed-alertedh]

A key detail that I hadn't thought about is that my test runner script needed to exit with a non-zero status code when any tests failed. Because I was running two sets of tests with the same bash script, that wasn't always the case if the second set of tests all passed.

I found a similar workaround to what @DalSoft suggests here: http://hermit.no/conditionally-fail-build-pull-requests-tfs-vsts/. In YAML the workaround from the linked blog post essentially looks like this (I opted to make it always fail, not just on PR's):

- script: 'echo 1>&2'
  failOnStderr: true
  displayName: 'If above is partially succeeded, then fail'
  condition: eq(variables['Agent.JobStatus'], 'SucceededWithIssues')

[brcrista]

This keeps coming up, so let me try to clarify the expected behavior here:

• Your test step should succeed if your tests succeed, and fail if any of your tests fail.
• Publish Test Results should succeed if the JUnit XML file (or whatever format it is) was successfully uploaded, and fail if uploading failed.
• Since the default behavior is to fail fast when a step fails, Publish Test Results must be marked as "Run always." Otherwise, it will get skipped if the test step fails.

I hear people suggesting that uploading failed tests with "Publish Test Results" should cause that step to fail (and therefore cause the build to fail). I would argue that failing the "Publish" step would be reporting the failure in the wrong place if the failure is actually in your test run. A failed test run should fail the build on its own. It shouldn't require "Publish Test Results" to detect the failure.

I admit, it might take a minute to figure out the "Run always" setting. But the intent of having build steps is to localize failures. Feel free to speak up if you still disagree, but I hope that helps clear things up.

[DalSoft]

@brcrista whilst technically correct as the publish hasn't failed the test task did - I think the confusion is because It's counter-intuitive to see green on the last build step if there is a failure. I expect to see red in the place that reports if my tests failed.

This is why the dotnet core task runs and publishes the tests together in one step.

[asinitson]

@brcrista: Technically correct does not equal useful.

It might be that some people were confused. For me there was never any confusion. But maybe I missed some kind of guideline in the spirit of "task should only fail with single condition"?

Let me reiterate the use case:

0. Your tests run outside of the Build Definition, so there is no return value you can use. In other words there is no test step Build Definition could know of.
1. In Build Definition the only task you have is "Publish Test Results" (in my case I just read a file from a file share).

The outcome I wish I could have:
Build is marked green when test results were read and test results are positive. I can have build results on the dashboard and if I see green I move on and I know that everything works.

Actual outcome:
Unless your test results are not published (something is badly broken) your build is always green. So if you use dashboard green color is kind of useless, since it just means "test results were read out", not "everything is working correctly". Every time you have to go in and check it out.

Now to the solution:

• "Publish Task Results" might have provided an option "Fail, if any tests failed". I don't think it contradicts the current behavior in any way. "Pass if publishing passed" just becomes "Pass if publishing passed AND if all tests passed".

• Having another built in task the sole purpose of which is failing the build depending on the test results. That would also work (although would seem a bit strange that you need to read out the test results file yet again).

• The actual solution I have (ugly! I wish I could use something built in instead, but produces the result I want)

1. The test runner (which runs outside of build definition) writes a file if all test pass called pass
2. After publishing test results there is command line task which looks for the file named pass, if file is not found it returns non-zero value.

I even thought about parsing the test result file and deciding on the outcome, but that seems especially ridiculous if you consider that "Publish Test Results" already parses test results and has all the information needed to signal pass/fail.

[brcrista]

Ok, I hear you. Right now, Publish Test Results doesn't parse the results files, it just uploads them to the service, which does the actual processing when it displays your results on the "Tests" tab (much like publishing artifacts):

That's why this suggestion would be a big change to the task.

I think the reason we haven't done the work for a special "Validate Test Results" task, or adding this functionality to Publish, is that most workflows do run and publish tests in the same job, and the step that runs the tests is usually the easiest and fastest place to fail the build. It already has the knowledge and no files need to get parsed.

It sounds like there are some workflows that want this step. Sorry to point you somewhere else, but https://developercommunity.visualstudio.com/ is where we'd like you to suggest this -- we prefer to use Issues to track tasks that not working as intended. Click "Suggest a feature:"

That said, here's a short PowerShell script that will find the number of failures in a JUnit XML (even works on hosted *nix agents 😉):

- powershell: |
    $junitXml = "junit/unit-test.xml"
    $(Get-Content $junitXml | Out-String) -match 'failures="(.*?)"'
    if ($matches[1] -eq 0)
    {
      Write-Host "No test failures"
    }
    else
    {
      # note that this will produce $LASTEXITCODE=1
      Write-Error "$($matches[1]) tests failed"
    }
  displayName: Check for test failures

[benken-parasoft]

Is this new feature live in VSTS? I'm not seeing the new checkbox. When will it be available?
I am referring to the new "Fail if there are test failures" checkbox that I see @vagisha-nidhi and @nigurr just landed.

[brcrista]

Yes, I meant this:

In YAML:

  - task: PublishTestResults@2
    inputs:
      testResultsFiles: '**/test-results.xml'
      testRunTitle: 'Python $(python.version)'
    condition: succeededOrFailed()

[nigurr]

@benken-parasoft, it will be available with next Azure Devops deployment (3 weeks away)

[divais]

Hi, it isn't live yet and will be available in few weeks as we have deployment freeze for the holiday season. Sorry for the delay.

[hisuwh]

Why has this been closed - this isn't available yet

[nigurr]

@hisuwh It's getting rolled out. It should be available within 2-3 days for everyone.

[hisuwh]

Ok thanks. You're documentation has already been updated which adds to the confusion

[hisuwh]

I'm using the YAML pipelines how will I know when I can use this feature?

[nigurr]

Please follow this - https://docs.microsoft.com/en-us/azure/devops/release-notes/
Since deployment goes stages by stages, everyone should be getting update within 2-3 weeks.

[Huffers]

Just found out a bunch of our tests had been failing for weeks but we didn't know because the build was green, hiding a bug in our production code. So infuriating.

Our build ran "dotnet test" a bunch of times manually in a PowerShell task because the tests need environment variables set, and the task ignored the exit codes of all but the last call to "dotnet test".

Good to see "Fail if there are test failures" appear as an option, but seems to just be inviting mess-ups by not having it checked by default. Maybe checked by default in the next version of the Publish Test Results task?

[nigurr]

@Huffers while we agree with "by default" it should be enabled, it breaks the compat scenario and cause more customer issues.
There could be existing pipelines which are already publishing results (with failed tests) however they might continue to run pipeline.