C# Parallel testing examples

[alexp1980]

Hello,

Do you have any examples of parallel tests in C#? I am using Nunit to run parallel tests but can't figure out how to split out the extent logs which are currently all going into one test name.

Regards

[bdewindt]

Hey Alex, I just got done developing a solution for a similar scenario. I don't know if it's the best solution, but it does generate a separate HTML report for each TextFixture class. The filename is a combination of the TestFixture class name and the current date/time.

Here's the base class:

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Runtime.CompilerServices;
using NUnit.Framework;
using NUnit.Framework.Interfaces;
using RelevantCodes.ExtentReports;


namespace NUnitExtentReporting
{
    internal class ExtentManager
    {
        public static ExtentReports Instance;

        private ExtentManager() { }

        /// <summary>
        /// Returns a ExtentReports object to be used for reporting
        /// </summary>
        /// <returns></returns>
        public static ExtentReports CreateExtentReportObject()
        {
            string reportFilename = string.Concat(DateTime.Now.ToString("yyyy-MM-dd-HHmm-ss"),"_", TestContext.CurrentContext.Test.ClassName, ".html");
            
            string fullPath = Path.Combine("C:\\ExtentReports\\, reportFilename);

            try
            {
                ExtentReports extent = new ExtentReports(fullPath, false);
		
		extent.X();	  //Use if reporting to an ExtentX server
                
                return extent;
            }
            catch(Exception e)
            {
		//Put some nifty logging here
            }

            return null;           
        }
    }

    /// <summary>
    /// Used to add Extent reporting capability to NUnit tests
    /// </summary>
    public class ExtentBase
    {
        protected internal ExtentReports Extent;
        protected internal ExtentTest ETest;

        /// <summary>
        /// One time setup method for the entire test run
        /// </summary>
        [OneTimeSetUp]
        [MethodImpl(MethodImplOptions.Synchronized)]
        public void FixtureInit()
        {
            ExtentManager.Instance = ExtentManager.CreateExtentReportObject();
            Extent = ExtentManager.Instance;
        }


        /// <summary>
        /// Setup method for individual tests
        /// </summary>
        [SetUp]
        [MethodImpl(MethodImplOptions.Synchronized)]
        public void TestSetup()
        {
            if (ExtentManager.Instance == null)
                return;

            string testName = TestContext.CurrentContext.Test.Name;

            string author = TestContext.CurrentContext.Test.Properties.ContainsKey("Author") 
                ? TestContext.CurrentContext.Test.Properties.Get("Author").ToString() 
                : null;

            List<string> categoryList = TestContext.CurrentContext.Test.Properties.ContainsKey("Category")
                ? TestContext.CurrentContext.Test.Properties["Category"].Cast<string>().ToList()
                : null;

            ETest = Extent.StartTest(testName);

            if (!string.IsNullOrEmpty(author))
                ETest.AssignAuthor(author);

            if (categoryList != null)
                ETest.AssignCategory(categoryList.ToArray());
        }

        /// <summary>
        /// Teardown method for individual tests
        /// </summary>
        [MethodImpl(MethodImplOptions.Synchronized)]
        [TearDown]
        public void TestTearDown()
        {
            if (ExtentManager.Instance == null)
                return;

            var status = TestContext.CurrentContext.Result.Outcome.Status;

            var message = string.IsNullOrEmpty(TestContext.CurrentContext.Result.Message)
                ? string.Empty
                : $"<pre>{TestContext.CurrentContext.Result.Message}</pre>";

            var stacktrace = string.IsNullOrEmpty(TestContext.CurrentContext.Result.StackTrace)
                    ? string.Empty
                    : $"<pre>{TestContext.CurrentContext.Result.StackTrace}</pre>";

            LogStatus logStatus;

            switch (status)
            {
                case TestStatus.Failed:
                    logStatus = LogStatus.Fail;
                    break;
                case TestStatus.Inconclusive:
                    logStatus = LogStatus.Warning;
                    break;
                case TestStatus.Skipped:
                    logStatus = LogStatus.Skip;
                    break;
                case TestStatus.Passed:
                    logStatus = LogStatus.Pass;
                    break;
                default:
                    logStatus = LogStatus.Unknown;
                    break;
            }

            string logText = $"Test [{TestContext.CurrentContext.Test.Name}] ended with a status of {logStatus}";

            if (logStatus == LogStatus.Fail)
                logText += $"<br><br>Message: {message}<br>Stacktrace: {stacktrace}";

            ETest.Log(logStatus, logText);

        }

        [MethodImpl(MethodImplOptions.Synchronized)]
        [OneTimeTearDown]
        public void FixtureTearDown()
        {
            if (ExtentManager.Instance == null)
                return;
				
            //Write the report
            try
            {
                Extent.Flush();
            }
            catch ()
            {
	        //Even more nifty logging
            }
        }
    }
}

Here's an example of a TestFixture class:

using System;
using NUnit.Framework;

namespace NUnitExtentReporting.Tests
{
    [Parallelizable(ParallelScope.Fixtures)]
    [TestFixture]
    public class RandomResultTests : ExtentBase
    {
        private static readonly Random ResultRng = new Random(1);
        private static readonly Random TimeRng = new Random(2);

        [Test]
        [Author("Quincy Magoo")]
        [Category("FirstCategory")]
        [Category("SecondCategory")]
        public void RandomResult01()
        {
            int r = ResultRng.Next(1, 10000);

            System.Threading.Thread.Sleep(TimeRng.Next(5000, 10000));

            if (r < 5000)
                Assert.IsTrue(true, "The test passed this time.");
            else
                Assert.IsTrue(false, "The test failed this time.");
        }
    }
}

Hope this helps!

[alexp1980]

Hi, thanks for the quick response. That's great but what I really wanted was to keep the same log file (i.e. keep a running file for a given day) and have the parallel tests listed separately within the file...

...but at the moment they are coming out within the first test name if running them in parallel...

Do you think this can be done at the moment?

Regards

[bdewindt]

I'm not sure it's possible with my approach above, but an NUnit addin would do the job. It would sit outside the scope of the TestFixtures., and allow you to build the report however you want so long as you handle the assigning of the child tests to the correct parent test (The TestFixture in this case) as they run. Take a look at https://github.com/nunit/docs/wiki/Event-Listeners if you're curious how the the event listeners in NUnit work, as it's what I used to get started.

I've been working on exactly such an approach for my own needs, but I again stress that I don't know if this is the best approach. I literally came up with this yesterday as a prototype for my company's reporting needs and I haven't cleaned it up yet, done extensive testing, etc. The only thing that isn't working right now is timestamps on the tests, but I think that may be Extent's fault and not mine. I'm still investigating it though. Also, this code is based on the latest Extent 3 alpha so the Extent approach is slightly different..

using NUnit.Engine;
using NUnit.Engine.Extensibility;
using System.Collections.Generic;
using System.Collections.Specialized;
using System.Xml;
using AventStack.ExtentReports;
using AventStack.ExtentReports.Reporter;

namespace NUnit.ExtentReportingAddin
{
    [Extension(Description = "Test Reporter Extension", EngineVersion = "3.4")]
    public class MyEventListener : ITestEventListener
    {
        public static ExtentReports Extent;

        public static Dictionary<string, ExtentTest> ParentTests = new Dictionary<string, ExtentTest>();
        public static Dictionary<string, ExtentTest> ChildTests = new Dictionary<string, ExtentTest>();

        public static Dictionary<string, string> CreatedParentsDictionary = new Dictionary<string, string>();
        public static Dictionary<string, string> NotCreatedParentsDictionary = new Dictionary<string, string>();

        public void OnTestEvent(string report)
        {
            XmlDocument xmlDoc = ProcessText(report);
            string eventType = xmlDoc.FirstChild.Name;

            switch (eventType)
            {
                case "start-run":
                    Extent = new ExtentReports();
                    var htmlReporter = new ExtentHtmlReporter("c:\\temp\\ExtentReport.html");
                    Extent.AttachReporter(htmlReporter);
                    break;

                case "start-suite":
                    if (xmlDoc.FirstChild.Attributes == null)
                        break;

                    NotCreatedParentsDictionary.Add(xmlDoc.FirstChild.Attributes["fullname"].Value, xmlDoc.FirstChild.Attributes["id"].Value);

                    break;

                case "start-test":
                    bool isParentCreated = false;

                    if (xmlDoc.FirstChild.Attributes == null)
                        break;

                    string testParentId = xmlDoc.FirstChild.Attributes["parentId"].Value;

                    foreach (KeyValuePair<string, string> pair in CreatedParentsDictionary)
                    {
                        if (pair.Value == testParentId)
                            isParentCreated = true;
                    }

                    if (!isParentCreated)
                    {
                        foreach (KeyValuePair<string, string> pair in NotCreatedParentsDictionary)
                        {
                            if (pair.Value != testParentId) continue;

                            ParentTests.Add(pair.Value, Extent.CreateTest(pair.Key));
                            CreatedParentsDictionary.Add(pair.Key, pair.Value);
                            NotCreatedParentsDictionary.Remove(pair.Key);
                            isParentCreated = true;
                            break;
                        }
                    }

                    if (isParentCreated)
                    {
                        foreach (KeyValuePair<string, ExtentTest> pair in ParentTests)
                        {
                            if (pair.Key == testParentId)
                                ChildTests.Add(xmlDoc.FirstChild.Attributes["id"].Value, pair.Value.CreateNode(xmlDoc.FirstChild.Attributes["name"].Value));
                        }
                    }

                    break;

                case "test-case":
                    if (xmlDoc.FirstChild.Attributes == null)
                        break;

                    string testId = xmlDoc.FirstChild.Attributes["id"].Value;
                    string result = xmlDoc.FirstChild.Attributes["result"].Value;
                    string author = string.Empty;
                    string[] categories = { };
                    string message = string.Empty;
                    string stacktrace = string.Empty;

                    foreach (KeyValuePair<string, ExtentTest> pair in ChildTests)
                    {
                        if (pair.Key != testId) continue;


                        foreach (XmlNode childNode in xmlDoc.FirstChild)
                        {
                            switch (childNode.Name)
                            {
                                case "properties":
                                    if (childNode.FirstChild.Attributes == null)
                                        break;
                                    NameValueCollection properties = new NameValueCollection();

                                    foreach (XmlNode entry in childNode.ChildNodes)
                                    {
                                        if (entry.Attributes == null)
                                            break;

                                        properties.Add(entry.Attributes["name"].Value, entry.Attributes["value"].Value);
                                    }

                                    string temp = properties["Author"];

                                    if (!string.IsNullOrEmpty(temp))
                                        author = temp;

                                    string[] temp2 = properties.GetValues("Category");

                                    if (temp2 != null)
                                        categories = temp2;
                                    break;
                                case "failure":
                                    foreach (XmlNode entry in childNode.ChildNodes)
                                    {
                                        switch (entry.Name)
                                        {
                                            case "message":
                                                message = entry.InnerText;
                                                break;
                                            case "stack-trace":
                                                stacktrace = entry.InnerText;
                                                break;
                                        }
                                    }
                                    break;
                            }
                        }

                        pair.Value.AssignCategory(categories);
                        pair.Value.AssignAuthor(author);

                        string logText = $"Test [{xmlDoc.FirstChild.Attributes["name"].Value}] ended with a status of {result}";

                        switch (result)
                        {
                            case "Passed":
                                pair.Value.Pass(logText);
                                break;
                            case "Failed":
                                logText += $"<br><br>Message: {message}<br><br>Stacktrace: {stacktrace}";
                                pair.Value.Fail(logText);
                                break;
                        }

                        ChildTests.Remove(pair.Key);

                        break;
                    }

                    break;

                case "test-suite":

                    break;

                case "test-run":
                    Extent.Flush();
                    break;

            }
        }
        public static XmlDocument ProcessText(string xmlText)
        {
            XmlDocument xmlDoc = new XmlDocument();
            xmlDoc.LoadXml(xmlText);

            return xmlDoc;
        }
    }
}

I know this isn't exactly what you're looking for, but the above approach should work for what you need with some tweaking. Currently it flushes the report once the entire test run is finished, but you could probably have it flush on child test completion instead so you can check in on the results throughout the day.

[alexp1980]

Hi,

Thanks, sounds intriguing. I'll have a play when I get time and see if I can use this how I need. Thanks again much appreciated.

[bdewindt]

You're most welcome :)

I'll try to remember to update this if I make any drastic changes or find a better way to do this.

[alexp1980]

Hi,

I'm not getting very far with this I'm afraid. If you ever come up with anything else please let me know.

There are some examples here of parallel runs in java but am not getting very far in seeing if that will convert to C#.

Regards

[bdewindt]

I'm not sure what kind of problem you're having. Using Extent through an NUnit addin is working well in my environment. I'm happy to help though if I can, just let me know what you're running into...

[alexp1980]

Hi, same as mentioned before. I have tests reporting with Extent Reports fines with Nunit sequentially, I just can't adjust the code to get the reports to work when tests run in parallel. If I try to run say two in parallel then I only get logs for one of them in Extent Reports.

[anshooarora]

I will add a few examples for parallel-runs with NUnit in the docs soon.

[alexp1980]

Thanks very much, I will continue to try in the mean time.

[anshooarora]

Btw, if you using version 2.40, there are a few examples here that you can check:

https://github.com/anshooarora/extentreports-csharp/tree/2.40.0/RelevantCodes.ExtentReports/RelevantCodes.ExtentReports.Tests/NUnit

Version 3+ examples will be added soon.

[alexp1980]

Thanks, yes I am using 2.41 at the moment. Those are good examples but I don't think they allow parallel tests to be logged into the same file. For example if wanted ManyCategories and ManyAuthors to be logged in the same file it only actually logs one of them.

Do you think it's possible to do this yet?