TestContext.Progress and TestContext.Error silently drop text that is not properly XML encoded

[KrzysFR]

I upgraded my test cases from NUnit 2.x to 3.x and replaced all occurrences of Console.WriteLine(...) to use either TestContext.Out, TestContext.Error or TestContext.Progress. After doing this, I started to see random parts of the test output missing.

After some investigation, I found that both TestContext.Progress and TestContext.Error do not work well with text that is not properly encoded XML.

Using NUnit 3.5, the following test only outputs the 1st and last line, because the 2nd and 3rd lines contain < and & that are not valid XML tokens (while > is allowed)

[Test]
public void Test_Non_Xml_Output()
{
	TestContext.Progress.WriteLine("Hello World");
	TestContext.Progress.WriteLine("Hello<World");
	TestContext.Progress.WriteLine("Hello&World");
	TestContext.Progress.WriteLine("Hello>World");
}

Output:

Hello World
Hello>World

If I use TestContext.Out.WriteLine(...) instead,I get the expected result once the test finishes.

Hello World
Hello<World
Hello&World
Hello>World

If I change the text to be manually XML encoded, then I get some output, although the literal has been changed to be the XML decoded version:

[Test]
public void Test_Non_Xml_Output()
{
	TestContext.Progress.WriteLine("Hello World");
	TestContext.Progress.WriteLine("Hello<World");
	TestContext.Progress.WriteLine("Hello&World");
	TestContext.Progress.WriteLine("Hello>World");
}

Output:

Hello World
Hello<World
Hello&World
Hello>World

This works, but then again the output is not what was written (especially troublesome for tests that validate proper encoding of custom text formats)

And if I switch to TestContext.Out I get the expected result:

Hello World
Hello<World
Hello&World
Hello>World

Are both TestContext.Progress and TestContext.Error only intended to send XML-friendly messages? Or are they supposed to write arbitrary literal text like TestContext.Out and Console.WriteLine(..) would ?

Looking at the code in TestOutput.ToXml(), it looks like the Text property is not encoded at all when generating the <test-output> XML fragment:

        /// <summary>
        /// Convert the TestOutput object to an XML string
        /// </summary>
        public string ToXml()
        {
            return TestName != null
                ? string.Format("<test-output stream='{0}' testname='{1}'>{2}</test-output>", Stream, TestName, Text)
                : string.Format("<test-output stream='{0}'>{1}</test-output>", Stream, Text);
        }

This would explain the issue above, and also would be open to abuse if someone would do something like TestContext.Progress.WriteLine("</<test-output><!--");. The Stream and TestName values should probably also be XML encoded, though in this case I would think they are safe (unless a C# method name can contain characters that require XML encoding?)

My guess is that the receiving end of the event will fail to decode the XML fragment, and silently drop the XmlException, hence not seeing anything in the output.

Side question: maybe if an instance of TestOutput is created for each line written, it should be a struct instead of a class, to recuce GC overhead?

[rprouse]

Excellent report, thanks. This should be a fairly easy fix, we just need to ensure that if we encode the output that it is getting decoded in the runners.

[KrzysFR]

If the decoder is using an XML parser, then decoding is probably already done automatically.

For the moment, I'm doing a very crude workaround by using the following helper method:

public static void Log(string msg)
{
    TestContext.Progress.WriteLine(msg.Replace("&", "&amp;").Replace("<", "&lt;"));
    //note: Replace(..) does not allocate a new string if the token is not present.
}

This is of course not a proper fix, but is sufficient for my test cases . Ideally, a proper XML encoder should be used instead, though I'm not sure if there is an efficient way, in the BCL, to do that without allocating a lot of memory...

[CharliePoole]

@KrzysFR I am unable to replicate the problem. All special XML characters display correctly for me.

Can you tell us how you are running your tests? NUnit3-console? NUnit3 VS adapter? etc.

[CharliePoole]

Additionally, please tell us what version of framework and runner you are using.

[KrzysFR]

I'm using mainly ReSharper 2016.2 in VS2015 which seem to have its own "test adapter" gui. I'm also testing using the latest version of TestDriven.NET which dumps the result in the output console, with same behavior.

I have installed the NuGet packages for NUnit 3.5.0, as well as the NUnit.ConsoleRunner 3.5.0 which I use with TeamCity for CI.

I don't have any other "NUnit Test Adapter" or equivalent extension installed in VS (apart from R# and TD.net)

[CharliePoole]

OK, I have replicated this using the NUnit 3.5 Console Runner. Oddly, it works correctly when running under NUnitLite, but with the console runner, the line disappears entirely as you reported. This leads me to suspect that the lack of encoding in TestOutput.ToXml that you pointed out is not the sole cause.

That said, I think the Text field should be a CDATA section, as it is for standard output that's included with the test result. The stream attribute is under NUnit's control and only takes on a set of discrete values. The testname may need some encoding since string arguments to methods may form part of the name.

I'll look further into the discrepancy between nunitlite and the console runner.

FYI, we normally don't work an issue that only shows up in third-party runners, which is why I was asking which NUnit-supplied runners you are using.

[CharliePoole]

Ah! Of course the reason for the discrepancy is obvious. There's no XML encoding / decoding involved when using NUnitLite because it references the framework directly.