Test Explorer - Give frameworks full control over test explorer layout

[thomhurst]

I assume that the test explorer view in IDEs is driven by the TestMethodIdentifierProperty on TestNodes?

This provides no way to show parameter types or argument types.

In data driven testing, I'd say it's not uncommon to inject in different objects that inherit from a base, and then want to run only those particular tests. This doesn't seem to be possible currently.

For example, consider a multi-tenanted application, and you want to run tests against a particular tenant you'd made changes for.

In pseudo code, the test is set up like this:

class TenantBase { ... }
class Tenant1 : TenantBase { ... }
class Tenant2 : TenantBase { ... }
class Tenant3 : TenantBase { ... }

[Inject<Tenant1>]
[Inject<Tenant2>]
[Inject<Tenant3>]
class MyTestClass(TenantBase tenant)

Meaning it is repeated for each tenant, with a different object being passed in.

Because the test explorer / test node properties have no visibility of the parameter types, or more importantly, the argument types, there is no way for it to display this in a nice way.

For example, in VSTest in NUnit, I could set up a test like this, and I'd get in the test explorer:

> MyTestClass
    > MyTestClass(MyNamespace.Tenant1)
    > MyTestClass(MyNamespace.Tenant2)
    > MyTestClass(MyNamespace.Tenant3)

This makes it super easy to selectively run the tests that I want.

---

I have tried this in my own framework.
Code

new TestMethodIdentifierProperty
(
...
TypeName: testDetails.ClassType.Name,
...
)

[ClassDataSource(typeof(Derived1))]
[ClassDataSource(typeof(Derived2))]
public class ClassDataSourceDrivenTests2
{
    private readonly Base _base;

    public ClassDataSourceDrivenTests2(Base @base)
    {
        _base = @base;
    }
    
    [Test]
    public void Base_Derived1()
    {
        // Dummy method
    }

    [Test]
    public void Base_Derived2()
    {
        // Dummy method
    }

    public class Base {}
    public class Derived1 : Base {}
    public class Derived2 : Base {}
}

Text explorer view:

So you can see that it makes it very difficult - I don't actually know which one I'm running.

Also, this would technically affect the test methods too - While they have the parameter types, there's no way to pass in the argument types, so we wouldn't be able to differentiate. Of course, we can mitigate this with a custom display name, but this isn't something we can do for the class itself.

[thomhurst]

Here's a current VSTest NUnit repro:

[TestFixtureSource(typeof(Base), nameof(Base.All))]
public class ClassDataSourceDrivenTests2
{
    private readonly Base _base;

    public ClassDataSourceDrivenTests2(Base @base)
    {
        _base = @base;
    }
    
    [Test]
    public void Base_Derived1()
    {
        Console.WriteLine(_base.GetType());
    }

    [Test]
    public void Base_Derived2()
    {
        Console.WriteLine(_base.GetType());
    }

    public class Base 
    {
        public static IEnumerable<Base> All()
        {
            yield return new Derived1();
            yield return new Derived2();
        }
    }
    
    public class Derived1 : Base {}
    public class Derived2 : Base {}
}

Displayed Test Explorer:
Rider:

Visual Studio:

[thomhurst]

I've tried setting the TypeName in TestMethodIdentifierProperty to more of a display name. That sort of works, but doesn't give you the same level of organisation of being able to group by class WITHOUT params and then WITH.

E.g.:

> MyClass
    > MyClass(Tenant1)
        - Test1
        - Test2
        - etc...
    > MyClass(Tenant2)
        - Test1
        - Test2
        - etc...
    > MyClass(Tenant3)
        - Test1
        - Test2
        - etc...

[thomhurst]

Thinking about this further, it'd be good to allow the framework to have full control over how tests display in test explorers. I think the best way to allow this would be a property that if provided, takes precedence over anything else (and simply falls back to existing behaviour if not provided)

That property can simply take a collection of paths. If any except the final path (because that evaluates to a single runnable test so then can't be expanded) are duplicated, then they will combine together in the test explorer.

For example:

public class TestExplorerDisplayProperty : IProperty
{
    public TestExplorerDisplayProperty(params string[] paths)
    {
        // ...
    }
}

And then in my data driven test, I could do:

new TestExplorerDisplayProperty
(
  [
    Test.Namespace,
    Test.BasicClassName,
    Test.ClassNameWithArgument,
    Test.BasicMethodName,
    Test.MethodNameWithArgument
  ]
)

Which would cause

> MyNamespace
  > MyClass
    > MyClass(Type1)
      > Test1
          Test1(Value1)
          Test2(Value2)
    > MyClass(Type2)
      > Test1
          Test1(Value1)
          Test2(Value2)

And for non-data driven tests I can just pass in fewer, more basic paths:

new TestExplorerDisplayProperty
(
  [
    Test.Namespace,
    Test.BasicClassName,
    Test.BasicMethodName,
  ]
)

Which would cause

> MyNamespace
  > MyClass
      Test1

[thomhurst]

In the existing NUnit example above, there's also differences between how Rider and Visual Studio display these. I think it's better to give this control to the testing framework rather than the IDE itself. (I know the IDE can obviously implement this behaviour however they like, but if there's a mechanism for the framework to control the layout it'd make sense for them to use it I think.) So this would fix differences/discrepencies between using different IDEs.

[Evangelink]

Thanks @thomhurst! This is something @MarcoRossignoli and I are trying to push since the inception of the new platform. Sadly Test Explorer is owned by a different team with different vision and priorities.

[thomhurst]

Feel free to send them this ticket if it'll help persuade them!

Currently the test explorer makes assumptions about how to organise tests. Different frameworks may have different intentions so giving that control can give a better experience for the user.

Heck, you could then even possibly allow users to group by test category or something in the test explorer. It could make things really interesting!

[Evangelink]

Currently the test explorer makes assumptions about how to organise tests. Different frameworks may have different intentions so giving that control can give a better experience for the user.

Entirely! Current Test Explorer enforces the shape and structure of everything when it should offer a common set of features but give flexibility to frameworks. The 4 levels is a huge limitation to me and I would like to be able to express my tests as it's done in F# Expecto where each group level is a "simple string" which leaves the user possibility to group tests functionally. This would also simplify things like BDD (e.g. SpecFlow).

[thomhurst]

Thanks for replying @Evangelink ! Completely agree with you. Hoping @MarcoRossignoli can throw some weight here too and maybe make this a possibility.

At work we have a multi-tenanted solution, and being able to run tests specifically for an injected class type (the tenant) would be a need for us. Much so that if went couldn't do it in the new world we'd just stick to existing VSTest NUnit. Hopefully that means something that there is a want for it and if we want to improve testing and give a framework that supports everything we need, the test explorer is part of that!

[thomhurst]

Also, you guys mention that the testing explorer is owned by another team, which I completely appreciate, but visual studio isn't the only testing explorer out there as you know, so is it a requirement to get approval from them first before implementing a generic mechanism that other tools could at least use?