Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[9.0-preview.2] Inconsistent messaging of success vs. warnings from dotnet test #39665

Closed
martincostello opened this issue Mar 21, 2024 · 5 comments · Fixed by dotnet/msbuild#9929
Closed

[9.0-preview.2] Inconsistent messaging of success vs. warnings from dotnet test #39665

martincostello opened this issue Mar 21, 2024 · 5 comments · Fixed by dotnet/msbuild#9929
Labels
Area-DotNet Test untriaged Request triage from a team member

Comments

@martincostello
Copy link
Member

Describe the bug

When tests are intentionally skipped, the .NET SDK reports inconsistent messaging on whether the build/tests succeeded. This behaviour also varies between running locally and in GitHub Actions (interactive vs. not).

.NET 8 
  Determining projects to restore...
  Restored /home/runner/work/alexa-london-travel/alexa-london-travel/test/LondonTravel.Skill.EndToEndTests/LondonTravel.Skill.EndToEndTests.csproj (in 733 ms).
  LondonTravel.Skill.EndToEndTests -> /home/runner/work/alexa-london-travel/alexa-london-travel/artifacts/bin/LondonTravel.Skill.EndToEndTests/release/LondonTravel.Skill.EndToEndTests.dll
Test run for /home/runner/work/alexa-london-travel/alexa-london-travel/artifacts/bin/LondonTravel.Skill.EndToEndTests/release/LondonTravel.Skill.EndToEndTests.dll (.NETCoreApp,Version=v8.0)
Microsoft (R) Test Execution Command Line Tool Version 17.9.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.27]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Cancel") [SKIP]
[xUnit.net 00:00:00.28]     Can_Invoke_Skill_And_Get_Valid_Response(content: "Alexa, ask London Travel about the Victoria line.") [SKIP]
[xUnit.net 00:00:00.29]     Can_Invoke_Skill_And_Get_Valid_Response(content: "Alexa, ask London Travel if there is any disruptio"···) [SKIP]
[xUnit.net 00:00:00.29]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "UnknownIntent") [SKIP]
[xUnit.net 00:00:00.29]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Status") [SKIP]
[xUnit.net 00:00:00.29]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Disruption") [SKIP]
[xUnit.net 00:00:00.29]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "SessionEnded") [SKIP]
[xUnit.net 00:00:00.29]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Stop") [SKIP]
[xUnit.net 00:00:00.29]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Launch") [SKIP]
  Skipped Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Cancel") [1 ms]
  Skipped Can_Invoke_Skill_And_Get_Valid_Response(content: "Alexa, ask London Travel about the Victoria line.") [1 ms]
  Skipped Can_Invoke_Skill_And_Get_Valid_Response(content: "Alexa, ask London Travel if there is any disruptio"···) [1 ms]
  Skipped Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "UnknownIntent") [1 ms]
  Skipped Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Status") [1 ms]
  Skipped Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Disruption") [1 ms]
  Skipped Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "SessionEnded") [1 ms]
  Skipped Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Stop") [1 ms]
  Skipped Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Launch") [1 ms]
[xUnit.net 00:00:00.33]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Help") [SKIP]
  Skipped Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Help") [1 ms]

Skipped! - Failed:     0, Passed:     0, Skipped:    10, Total:    10, Duration: 49 ms - LondonTravel.Skill.EndToEndTests.dll (net8.0)
.NET 9 in CI 
  Determining projects to restore...
  Restored /home/runner/work/alexa-london-travel/alexa-london-travel/test/LondonTravel.Skill.EndToEndTests/LondonTravel.Skill.EndToEndTests.csproj (in 948 ms).
/usr/share/dotnet/sdk/9.0.100-preview.2.24157.14/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(314,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/home/runner/work/alexa-london-travel/alexa-london-travel/test/LondonTravel.Skill.EndToEndTests/LondonTravel.Skill.EndToEndTests.csproj]
  LondonTravel.Skill.EndToEndTests -> /home/runner/work/alexa-london-travel/alexa-london-travel/artifacts/bin/LondonTravel.Skill.EndToEndTests/release/LondonTravel.Skill.EndToEndTests.dll
.NET 9 Locally 
Restore complete (0.7s)
You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy
  LondonTravel.Skill.EndToEndTests succeeded (0.4s) → artifacts\bin\LondonTravel.Skill.EndToEndTests\release\LondonTravel.Skill.EndToEndTests.dll
  LondonTravel.Skill.EndToEndTests test succeeded with warnings (2.0s)
    C:\Program Files\dotnet\sdk\9.0.100-preview.2.24157.14\Microsoft.TestPlatform.targets(46,5): warning : [xUnit.net 00:00:00.36]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Cancel") [SKIP] [C:\Coding\martincostello\alexa-london-travel\test\LondonTravel.Skill.EndToEndTests\LondonTravel.Skill.EndToEndTests.csproj]
    C:\Program Files\dotnet\sdk\9.0.100-preview.2.24157.14\Microsoft.TestPlatform.targets(46,5): warning : [xUnit.net 00:00:00.37]     Can_Invoke_Skill_And_Get_Valid_Response(content: "Alexa, ask London Travel about the Victoria line.") [SKIP] [C:\Coding\martincostello\alexa-london-travel\test\LondonTravel.Skill.EndToEndTests\LondonTravel.Skill.EndToEndTests.csproj]
    C:\Program Files\dotnet\sdk\9.0.100-preview.2.24157.14\Microsoft.TestPlatform.targets(46,5): warning : [xUnit.net 00:00:00.37]     Can_Invoke_Skill_And_Get_Valid_Response(content: "Alexa, ask London Travel if there is any disruptio"···) [SKIP] [C:\Coding\martincostello\alexa-london-travel\test\LondonTravel.Skill.EndToEndTests\LondonTravel.Skill.EndToEndTests.csproj]
    C:\Program Files\dotnet\sdk\9.0.100-preview.2.24157.14\Microsoft.TestPlatform.targets(46,5): warning : [xUnit.net 00:00:00.37]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "UnknownIntent") [SKIP] [C:\Coding\martincostello\alexa-london-travel\test\LondonTravel.Skill.EndToEndTests\LondonTravel.Skill.EndToEndTests.csproj]
    C:\Program Files\dotnet\sdk\9.0.100-preview.2.24157.14\Microsoft.TestPlatform.targets(46,5): warning : [xUnit.net 00:00:00.37]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Status") [SKIP] [C:\Coding\martincostello\alexa-london-travel\test\LondonTravel.Skill.EndToEndTests\LondonTravel.Skill.EndToEndTests.csproj]
    C:\Program Files\dotnet\sdk\9.0.100-preview.2.24157.14\Microsoft.TestPlatform.targets(46,5): warning : [xUnit.net 00:00:00.37]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Disruption") [SKIP] [C:\Coding\martincostello\alexa-london-travel\test\LondonTravel.Skill.EndToEndTests\LondonTravel.Skill.EndToEndTests.csproj]
    C:\Program Files\dotnet\sdk\9.0.100-preview.2.24157.14\Microsoft.TestPlatform.targets(46,5): warning : [xUnit.net 00:00:00.37]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "SessionEnded") [SKIP] [C:\Coding\martincostello\alexa-london-travel\test\LondonTravel.Skill.EndToEndTests\LondonTravel.Skill.EndToEndTests.csproj]
    C:\Program Files\dotnet\sdk\9.0.100-preview.2.24157.14\Microsoft.TestPlatform.targets(46,5): warning : [xUnit.net 00:00:00.37]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Stop") [SKIP] [C:\Coding\martincostello\alexa-london-travel\test\LondonTravel.Skill.EndToEndTests\LondonTravel.Skill.EndToEndTests.csproj]
    C:\Program Files\dotnet\sdk\9.0.100-preview.2.24157.14\Microsoft.TestPlatform.targets(46,5): warning : [xUnit.net 00:00:00.37]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Launch") [SKIP] [C:\Coding\martincostello\alexa-london-travel\test\LondonTravel.Skill.EndToEndTests\LondonTravel.Skill.EndToEndTests.csproj]
    C:\Program Files\dotnet\sdk\9.0.100-preview.2.24157.14\Microsoft.TestPlatform.targets(46,5): warning : [xUnit.net 00:00:00.41]     Can_Invoke_Intent_Can_Get_Json_Response(payloadName: "Help") [SKIP] [C:\Coding\martincostello\alexa-london-travel\test\LondonTravel.Skill.EndToEndTests\LondonTravel.Skill.EndToEndTests.csproj]

Build succeeded with warnings in 3.8s
Test run succeeded. Total: 10 Failed: 0 Passed: 0 Skipped: 10, Duration: 2.0s

In CI there's no detail of the skipped tests (or indeed that any tests were even run).

Locally, the messaging is inconsistent (emphasis mine):

  • LondonTravel.Skill.EndToEndTests succeeded
  • LondonTravel.Skill.EndToEndTests test succeeded with warnings
  • Build succeeded with warnings in 3.8s
  • Test run succeeded. Total: 10 Failed: 0 Passed: 0 Skipped: 10, Duration: 2.0s

The messaging suggests that build and test both succeeded and succeeded with warning.

Based on my intent at least, in this scenario I would expect the build and the tests to both be successful because no tests failed.

If regardless of my intent skipping should be treated as a warning, then then messaging should be consistent and arguably it should be possible to treat skipped tests as not being at a warning level.

To Reproduce

  1. Clone martincostello/alexa-london-travel@1152660
  2. Run build.ps1 from the root of the repository.

Further technical details

.NET SDK 9.0.100-preview.2.24157.14

/cc @nohwnd
@dotnet-issue-labeler dotnet-issue-labeler bot added Area-DotNet Test untriaged Request triage from a team member labels Mar 21, 2024
@martincostello
Copy link
Member Author

martincostello commented Mar 21, 2024 •
edited
Loading

The CI vs. not thing might actually be a different issue.

Carrying on with what I was actually doing when I observed the original issue, I find that in CI the tests failing isn't being reported on the console - I just get /usr/share/dotnet/sdk/9.0.100-preview.2.24157.14/Microsoft.TestPlatform.targets(46,5): error MSB6006: "dotnet" exited with code 1. [/home/runner/work/alexa-london-travel/alexa-london-travel/test/LondonTravel.Skill.EndToEndTests/LondonTravel.Skill.EndToEndTests.csproj].

I can only see why the tests failed with the GitHub Actions test logger.

@nohwnd
Copy link
Member

nohwnd commented Mar 22, 2024

I will fix net9 output when all tests are skipped to say Skipped! rather than Passed!.

The CI output is interesting, you are using the preview2 version which should print errors via msbuild api, so they should show up on console, maybe it is the extra logger the action is providing, but it seems unlikely.

Most of my updates on terminal logger integration go here: microsoft/vstest#4843

@nohwnd
Copy link
Member

nohwnd commented Mar 26, 2024

I've re-read this again, and looked at terminal logger code, and I think it will be for the best if the test summary in terminal logger does not report success or fail, but instead just says: Test summary: followed by test counts. All error that are reported will be accounted for in the result of the build, as well as all warnings, so repeating the status is only asking for inconcistencies.

@nohwnd nohwnd mentioned this issue Mar 26, 2024
Merged
@martincostello
Copy link
Member Author

The CI output is interesting, you are using the preview2 version which should print errors via msbuild api, so they should show up on console, maybe it is the extra logger the action is providing, but it seems unlikely.

Observing this still with preview.3 - here's an example.

@baronfel
Copy link
Member

We couldn't get this into the preview 3 cutoff, but the PR is undergoing approvals and should be in preview 4.

@martincostello martincostello mentioned this issue May 31, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Area-DotNet Test untriaged Request triage from a team member
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove result indicator from test summary dotnet/msbuild
3 participants
@baronfel @martincostello @nohwnd