-
Notifications
You must be signed in to change notification settings - Fork 640
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Console.WriteLine does not appear to be working in unit tests #406
Comments
I specifically brought this to the attention of the NUnit team when we started targeting NUnit 3 instead of NUnit 2. There seems to be more options in NUnit 3, but I haven't worked out which one is the "right" one for our use cases
I believe all of the mobile platforms have a standard way to write output, but they would need further investigation. Verbose SettingDo note that we also had an issue with some of the tests with larger output filling up a buffer in Visual Studio and crashing the IDE. Those tests have since had their Currently verbose is set to true when running tests in {
"tests": {
"verbose": "true"
}
} Abstraction of Output MethodsI suggest using an abstract set of Do note that in some cases, output is written to an Future PortingOne additional thing to keep in mind - when porting future tests, the logical thing to do is to change
Whatever syntax we actually end up with, we should strive to make these lines intuitive to convert to .NET. |
Thanks @NightOwl888, great info. I've done many hours of research on this and here is what I have found. I'd really value your thoughts. Console.WriteLine IS going somewhereAlmost by accident, I discovered that at least for failing tests the output for At first glance it seems like it's just telling that there was a certain type of exception. BUT if you right click that failing test and click "Copy Details" it will place the WHOLE message on the clipboard. By pasting that in some editor, I came to realize that the exception message is crazy long, and in fact does contain the While this does not send the Other API OptionsI have tried all of theses:
In the version of NUnit that Lucene.Net is using, none of these go to the output window. Although, Parallel Unit testsOne of the reason that this whole issue cropped up in NUnit3 which did not exist in NUnit2 is that in NUnit3 there was a concern that when running unit tests in parallel, the output from something like The idea was to have some methods on that class write immediately (somewhere?!) and to have some be buffered at the unit test level and only output with the test results when that unit test completes. This in theory sounds pretty smart to me, as it gets away from the whole problem of having output intermixed for multiple tests but still provides the ability to theoretically output in real time if desired. In practice, the actual implementation seems wonky, however. Here is what my tests show based on examining the exception message content, and remember I can get none of these to go to the output window (or intermediate window) currently:
In NUnit 3.17 it's now possible to send Console.WriteLine to Output WindowApparently this is a recognized NUnit3 issue, issue 343. The issue was first placed in June 2017 and got plenty of community banter. The issue was resolved with the release of NUnit3TestAdapter ver 3.17 which is the current production release of NUnit. This is to say that being able to have Currently Lucene.Net uses NUnit3TestAdapter ver 3.16.1. Wrap-up and Call for FeedbackSo it seems tempting to recommend that we upgraded Lucene.net from NUnit3.16.1 to NUnit 3.17 but I know in a project of this size, and with the number of unit test it has, there may be hidden issues I'm unaware of. The good news is whether we upgrade NUnit or now, we now know how to see the output from As an aside we now have a way to make the random seed for the test visible when a test fails. We can just use But we do still have to figure out if upgrading to NUnit 3.17 is worth while. Given that I don't personally use console messages that much in my own approach to debugging, I don't have a strong opinion either way, especially since I now know how to see the output in the exception messages. What are your thoughts? |
There is one issue: Xamarin tests (which I am currently doing work on) cannot use anything higher than NUnit However, the way forward is to downgrade to that version on It is becoming clear that the I will need to check how difficult it may be to tie the output of Parallel Unit testsSounds good in theory, but in Lucene.NET we are limited to running tests serially due to shared static members in the test framework. Changing this could be done, but would require some fairly extensive refactoring with how the codecs are loaded and breaking API changes. So, in effect we are gaining nothing from this NUnit "feature" except confusion about where the results are and even worse, limits that cause truncation of the output and/or crashes. Here is a test that is crashing when the buffer in VS is overloaded. VS Test Output"Copy Details" seems to have some finite limit. If the amount of text exceeds the limit, it is truncated and may not be very useful. Unfortunately, the Lucene tests were designed with an infinite amount of space to log into. I don't find the logging here particularly useful either. However, it is not just meant for the Lucene.NET team, it is meant for anyone that consumes the test framework - therefore, it should be set up in a reasonable way to make debugging easy for those who depend on it. |
I kinda thought that might be the case. I see no pressing reason to switch from serially running the tests. That keeps our lives more sane.
If we go to with NUnit 3.17 then the full output of
Is it available elsewhere where I could take a look at it just to better understand what such a test runner looks like? |
I have setup a temporary repo so you can have a peek.
|
I'm unable to locate the |
|
I totally didn't see that the whole project was there. Thank you! |
This issue can be assigned to me. I have confirmed that upgrading to NUnit3TestAdapter 3.17.0 does in fact fix the issue with However, I'm still not sure if this gonna be a go or no go as running the tests occasionally blows up NUnit, and other times blows up Visual Studio all together. I see that NUnit itself versions independently of NUnit3TestAdapter . Lucene.Net currently uses NUnit 3.12.0 so I I may try to upgrade it to the latest version which is NUNit 3.13.1 to see if that fixes the stability issues I'm seeing with the upgrade NUnit3TestAdapter 3.17.0 |
Great. I have assigned the issue to you. Thanks for putting in the grunt work on this. When using NUnit 2.x, we didn't have any issues running out of space when the output actually went to the output window. That problem began occurring when NUnit 3.x started putting the output in each test, which apparently has a limited amount of space available. Maybe there is a way to increase the space available, but it certainly is a lot more like the intended experience direct the console output somewhere it could be viewed in one chunk of text, as there is information that is output between each test run that may otherwise be lost. |
Upgrading the two as a pair seemed to make things happy. Pull request submitted. PR#412 |
Currently Lucene.Net uses
Console.WriteLine
1804 times (although some lines are commented out). A quick scan of the files containingConsole.WriteLine
shows that it's being used in unit tests petty much exclusively. As a new developer on the project looking into a failing test, the first thing I wanted to see was theConsole.WriteLine
output for the failing test. But I struggled to find where the output was showing up and after talking with @NightOwl888 I came to understand that this is something that still needs to be resolved. CurrentlyConsole.WriteLine
does not appear to be working in unit tests.@NightOwl888 also mentioned that according to How do I get nunit3-console to output my debug to screen (on a Windows box)?, we should maybe be using
TestContext.WriteLine()
instead. So this is something I'm now investigating.The text was updated successfully, but these errors were encountered: