Failing tests don't fail build when warningsAsErrors is false

[ryanbrandenburg]

We've started setting warningsAsErrors to false in our builds of Arcade projects because that's what we did before, and we have some warnings that would currently cause us to start failing, but when we do this test failures don't fail the build because they're only logged as warnings. I suspect this is caused by this line, where LogStandardErrorAsError="false". I've noticed that that's set that way in a couple different Exec's, so possibly this is not the only area affected.

Repro:

1. Clone git@github.com:aspnet/Extensions.git
2. git checkout rybrande/Arcade
3. .\build.cmd

That takes a while to run, but if I'm right it should replicated against any arcade repo with a failing test against which warnaserrors=false was passed.

[ryanbrandenburg]

Here is an example of this in the wild.

CC @markwilkie

[tmat]

This line should fail the build:
https://github.com/dotnet/arcade/blob/master/src/Microsoft.DotNet.Arcade.Sdk/tools/XUnit/XUnit.targets#L112

[ryanbrandenburg]

That line is output, but as a warning. You can see it at the very bottom of the log I linked above.

[natemcmaster]

+1 to what Ryan said. It's clearly a bug.

[nguerrera]

It's behaving as though there's a ContinueOnError=WarnAndContinue in effect when that Error is logged.

[nguerrera]

@rainersigwald

[tmat]

Yeah, seems like a bug in msbuild to me.

[markwilkie]

Next step is to find and reference the tracking msbuild bug (assuming there is one)