New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing output from console, diagnostics and ITestOutputHelper #718

Closed
ghost opened this Issue Dec 21, 2015 · 5 comments

Comments

Projects
None yet
1 participant
@ghost

ghost commented Dec 21, 2015

There are major problems with output missing from running xunit 2.1 in both Visual Studio 2015 and outside Visual Studio when using xUnit.net DNX Runner 64-bit DNX 4.5.1 (ASP.NET 5 RC1 + Update 1).

From a test I write output using Console.WriteLine, System.Diagnostics.Debug.WriteLine and ITestOutputHelper.WriteLine. Hence I expect to see 3 different outputs when running the test.

Bug 1: When running the test inside visual studio I get NO output whatsoever !!!
Bug 2: When running in command line I get only the console output (1/3 outputs).
Bug 3: When running in command line and exporting to xml, I get only the ITestOutputHelper.WriteLine outpit (1/3 outputs).

@bradwilson

This comment has been minimized.

Show comment
Hide comment
@bradwilson

bradwilson Dec 21, 2015

Member
  1. Open a bug with the Visual Studio test team
  2. We don't capture any form of output in v2 (see the release notes section titled "Test Output": http://xunit.github.io/release-notes/2015-03-16.html)
  3. This is expected behavior, per the release notes
Member

bradwilson commented Dec 21, 2015

  1. Open a bug with the Visual Studio test team
  2. We don't capture any form of output in v2 (see the release notes section titled "Test Output": http://xunit.github.io/release-notes/2015-03-16.html)
  3. This is expected behavior, per the release notes
@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Dec 22, 2015

Where exactly should I open the bug? I looked at "xunit/visualstudio.xunit" and it refers to this project for bugs?
Why does scenario 2 (running in command line) not write results for ITestOutputHelper.WriteLine?

As for "this is expected behavior", I do understand the technical reasons but.... It violates the principle of not surprising users and it adds additional coupling between unit tests and the particular test framework which is bad.

For tests that do not need to run parallel it is very, very annoying (+ surprising) that console/diagnostics input is not captured. There should be an user option for this and capturing should only be disabled for parallel tests by default

ghost commented Dec 22, 2015

Where exactly should I open the bug? I looked at "xunit/visualstudio.xunit" and it refers to this project for bugs?
Why does scenario 2 (running in command line) not write results for ITestOutputHelper.WriteLine?

As for "this is expected behavior", I do understand the technical reasons but.... It violates the principle of not surprising users and it adds additional coupling between unit tests and the particular test framework which is bad.

For tests that do not need to run parallel it is very, very annoying (+ surprising) that console/diagnostics input is not captured. There should be an user option for this and capturing should only be disabled for parallel tests by default

@bradwilson

This comment has been minimized.

Show comment
Hide comment
@bradwilson

bradwilson Dec 22, 2015

Member

Where exactly should I open the bug?

https://connect.microsoft.com/visualstudio

Why does scenario 2 (running in command line) not write results for ITestOutputHelper.WriteLine?

Output is only written to the XML (and reports) at this time. You can open a feature request to have it output to the screen in addition to the XML.

It violates the principle of not surprising users

As opposed to unexpectedly capturing something you may be testing or capturing yourself? Parallelization issues aside, there is no right answer here, and when there's no right answer, we tend towards leaving things to their original purpose. It was a mistake for us to capture this output in v1, just as it was a mistake to add [ExpectedException] to NUnit. We fix those mistakes when we can. :)

Member

bradwilson commented Dec 22, 2015

Where exactly should I open the bug?

https://connect.microsoft.com/visualstudio

Why does scenario 2 (running in command line) not write results for ITestOutputHelper.WriteLine?

Output is only written to the XML (and reports) at this time. You can open a feature request to have it output to the screen in addition to the XML.

It violates the principle of not surprising users

As opposed to unexpectedly capturing something you may be testing or capturing yourself? Parallelization issues aside, there is no right answer here, and when there's no right answer, we tend towards leaving things to their original purpose. It was a mistake for us to capture this output in v1, just as it was a mistake to add [ExpectedException] to NUnit. We fix those mistakes when we can. :)

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Dec 22, 2015

Thanks for the details.

All other test frameworks that I have used (and I have used many on both .NET and Java) captures console/diagnostics output so if it was a mistake in v1 then all other frameworks has made this mistake as well. So I think it is very safe to say this new behavior is surprising.

But yes, I agree that capturing the wrong stuff is wrong too. Could maybe be solved with options to have tests run single-threaded with capturing on + some warnings if there is output that is being suppressed (+ write ITestOutputHelper output to screen).

ghost commented Dec 22, 2015

Thanks for the details.

All other test frameworks that I have used (and I have used many on both .NET and Java) captures console/diagnostics output so if it was a mistake in v1 then all other frameworks has made this mistake as well. So I think it is very safe to say this new behavior is surprising.

But yes, I agree that capturing the wrong stuff is wrong too. Could maybe be solved with options to have tests run single-threaded with capturing on + some warnings if there is output that is being suppressed (+ write ITestOutputHelper output to screen).

@bradwilson

This comment has been minimized.

Show comment
Hide comment
@bradwilson

bradwilson Nov 10, 2017

Member

Closed for age.

Member

bradwilson commented Nov 10, 2017

Closed for age.

@bradwilson bradwilson closed this Nov 10, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment