Use array argument contents in name of parameterized tests rather than just array type.

[yaakov-h]

TestCase and TestCaseSource make it too easy to create tests with duplicate names.

Case 1: Arrays

Arrays are not formatted further, only their type name becomes part of the test name by default. For example, given the following test fixture:

using NUnit.Framework;

namespace NUnitDuplicateTests
{
    public class DuplicateTestFixture
    {
        [TestCase((object)new[] { "John", "Smith" })]
        [TestCase((object)new[] { "Emperor", "Palpatine" })]
        [TestCase((object)new[] { "Groot" })]
        public void EverybodyHasAtLeastTwoNames(string[] nameParts)
        {
            Assert.That(nameParts, Has.Length.GreaterThanOrEqualTo(2));
        }
    }
}

Only one test is discovered: NUnitDuplicateTests.DuplicateTestFixture.EverybodyHasAtLeastTwoNames(System.String[]).

Only one test shows up in the Visual Studio test explorer.

Only one unique test shows up with nunit3-console --explore, but is printed 3 times.

This creates issues for applications that use the Explore API, if they wish to track each test individually for various purposes.

I would expect the test name to be something along the lines of NUnitDuplicateTests.DuplicateTestFixture.EverybodyHasAtLeastTwoNames(["Emperor", "Palpatine"]).

Case 2: Test Names

I would logically expect a test name from TestCaseAttribute or TestCaseData to be local to the method that declares the test. It is often convenient to use the same test data for multiple test methods, e.g.

using System.Collections;
using NUnit.Framework;

namespace NUnitDuplicateTests
{
    public class DuplicateTestFixture
    {
        [TestCaseSource(nameof(Names))]
        public void EverybodyHasAtLeastTwoNames(string[] nameParts)
        {
            Assert.That(nameParts, Has.Length.GreaterThanOrEqualTo(2));
        }

        [TestCaseSource(nameof(Names))]
        public void EverybodyIsOnTheImperialPayroll(string[] nameParts)
        {
            // TODO: Talk to the Payroll division on Coruscant.
        }

        public static IEnumerable Names
        {
            get
            {
                yield return new TestCaseData(new[] { "Emperor", "Palpatine" }).SetName("Our grand Emperor");
                yield return new TestCaseData(new[] { "Darth", "Vader" }).SetName("Luke's Dad");
            }
        }
    }
}

Although this fixture declares four tests, there are only two unique test names - NUnitDuplicateTests.DuplicateTestFixture.Our grand Emperor and NUnitDuplicateTests.DuplicateTestFixture.Luke's Dad.

I would expect there to be four, somewhat resembling the following:

1. NUnitDuplicateTests.DuplicateTestFixture.EverybodyHasAtLeastTwoNames."Our grand Emperor"
2. NUnitDuplicateTests.DuplicateTestFixture.EverybodyHasAtLeastTwoNames."Luke's Dad"
3. NUnitDuplicateTests.DuplicateTestFixture.EverybodyIsOnTheImperialPayroll."Our grand Emperor"
4. NUnitDuplicateTests.DuplicateTestFixture.EverybodyIsOnTheImperialPayroll."Luke's Dad"

Case 3: String truncation

String parameters are truncated to 40 characters, which makes it difficult to use long strings (such as URLs) in test cases. e.g.:

using NUnit.Framework;

namespace NUnitDuplicateTests
{
    public class DuplicateTestFixture
    {
        [TestCase("The Imperial Death Star is a collossal waste of money. The Empire could better use those funds on supplies and soldiers.")]
        [TestCase("The Imperial Death Star is a collossal monument to the power of the Empire.")]
        public void AllOpinionsAreValid(string opinion)
        {
            Assert.That(opinion, Is.Not.Null, "Not having an opinion is not in itself an opinion.");
        }
    }
}

This fixture only has one unique test name, NUnitDuplicateTests.DuplicateTestFixture.AllOpinionsAreValid("The Imperial Death Star is a collossa...").

From this, I see three resolutions:

1. Can test names be made smarter by introspecting arrays?
2. Can explicit test names be made local to their method, not local to their fixture?
3. Can strings be not truncated, or the truncation limit be increased well beyond 40 characters?

[yaakov-h]

For the record, I have managed to solve all 3 in my own project by subclassing TestCaseAttribute and TestCaseSourceAttribute, but:

1. This requires that I create my own implementation of every test case builder shipped with nunit.framework. I haven't rebuilt ValueSourceAttribute, for example.
2. This unfortunately requires either complete duplication of the NUnit attribute classes, or subclassing and using reflection to access the internal field TestMethod.params. I haven't found another way to do this.

Is this something that NUnit itself can handle?

[CharliePoole]

It's fairly well-known that NUnit neither requires nor creates unique test names. As a user, you can set the name of every testcase to "MyTest" and NUnit will happily run them. Of course, this raises the problem that the test then becomes very difficult to identify in the output.

You propose three improvements:

1. "Introspecting arrays" By this do you mean that the array should be represented by listing it's contents?
2. "Explicit test names made local to their methods" By specifying template arguments in the name, this is already possible, e.g.: "{m}_Some_Suffix"
3. "Strings not truncated" "{m}{a}" or "limit increased" "{m}{a:100}

So of the three - and assuming I understand what you are suggesting - the first is a possible change, while the other three are already available.

[yaakov-h]

1. Yes

2, 3.: Wait, what? Where is this documented? I haven't heard of this feature before.

[yaakov-h]

Looks like {m}{a} is still truncated, but {m}{a:100} does increase the truncation limit.

[CharliePoole]

The specification we worked from is here: https://github.com/nunit/dev/wiki/Generation-of-Test-Names

It should have been transfered to the documentation wiki! I'll make a note of that.

I'll have to look more closely at the code to see what {a} is supposed to do - I was referring to it from memory.

[yaakov-h]

Note that in this usage, it will generally only make sense to use {m}, {a} or {o} specifiers. However, NUnit will use whatever is provided.

{o} isn't mentioned anywhere else on that page - do you recall what this one does?

[CharliePoole]

I think it's an older planned format, replaced by {0} through {9}

I don't think all of this is implemented. I'll have to figure that out when I copy some of it to the docs. Code is at https://github.com/nunit/nunit/blob/master/src/NUnitFramework/framework/Internal/TestNameGenerator.cs

[rprouse]

@CharliePoole should we mark this as a documentation issue, or is there any coding we should consider for this?

[CharliePoole]

I created nunit/docs#38 for updating the docs. The issue of uniqueness may require more work but it's actually covered in several existing issues.

[rprouse]

I am going to close then.

[yaakov-h]

What about array introspection (1)?

[CharliePoole]

By that do you mean the use of the content of an array in it's representation as an argument?

I think we have that as an issue somewhere, but I'm unable to find it!

[yaakov-h]

Yep. Is it worth creating/duplicating then?