Test Explorer sorting

[kendrahavens]

Description

This issue exists so the community can leave feedback on the sorting and/or grouping of tests in the Test Explorer.

Details

The Visual Studio 2017 Update 15.6 Preview introduces the hierarchy view to the Test Explorer. This first iteration of the hierarchy organizes tests by Project, Namespace, Class, and theory/data driven tests where applicable. Future improvement should include a configurable hierarchy where users can choose the ordering they wish (such as Project, Class, then Duration).

This introduces a few design pivots we would like feedback on. The hierarchy view sorts tests alphabetically as opposed to outcome. Alternatively, the traditional list view retains the benefit of tests being sorted by outcome. Tests quickly bubble up in their appropriate groupings to grab your attention. The different group by options are shown below for comparison.

Since the hierarchy view is extremely useful for navigation, we prioritized a sorting order that would stay static despite tests having different outcomes between test runs. The benefit is you always know where to look in the list for a certain test. The benefit of tests being sorted by outcome is it quickly bubbles up the failing tests in the list that most need your attention.

Call to Action

Please weigh in. What are your preferred patterns for test grouping and sorting?

[rrector]

The non-hierarchal "Group by traits" view now just seems to be in a random order. Please bring back the sorted by outcome presentation for this.

Ideally, there would be a "Sort by" menu that has the same options as the "Group by" to allow people to lay things out in the way that works best for them.

[kendrahavens]

@rrector Thanks for the feedback! I'll look into why the sorting in the regular groupby might have changed.

I like the idea of adding a "Sort by" menu. If you have any ideas of what UI would work well for that please share! It could be a separate button, but I don't want to add too many buttons to the menu if we can help it. Sorting options could be added to the context menu that appears when one clicks on the "Group by" button.

I'm wondering if users would want to save the viewing options selected for different test sets. For example, for performance tests users may always want them ordered by duration, or for functional tests the sorting would always need to be by an ordered list. Maybe it could be a property saved in a .runsettings file that VS can read to know what order the tests should be displayed.

[ledped]

Does anyone know where the metadata that is used to sort according to the different "Sort by" options is stored?
My teammates and I are getting very different outcome, when sorting by the same "Sort by" and even different quantities of tests, when using the Test Explorer, although we are cloning the same Git Repo.

[kendrahavens]

@ledped That sounds like a bug. Feel free to open an issue on developer community. A repro solution with the branch you are seeing this on is always super helpful!

(Sorry to redirect you there. This issue was meant more for design discussion.)

The metadata used to sort depends on what test framework you are using. In the bug, could you provide what test framework you are using and what sortings are causing inconsistent results? The sortings on Project, Namespace, and Class are decided by the Fully Qualified Name assigned to the test. Other options like traits or categories are decided by the test framework.

[tvaloy]

I use grouping by outcome. In this grouping the failed tests are on top and successful tests are right below. In earlier versions of VS2015 recently executed tests were always placed at the top of the list of successful tests. This made it very easy to navigate to the last run tests. Now these tests are ordered alphabetically with all other tests. In solutions with hundreds of tests this makes it hard to find the recently run tests.

Please make the old ordering an option.

[kendrahavens]

Thank you for the suggestion @tvaloy ! I've opened an issue to track interest in adding an option to sort by recently run tests in the Test Explorer groupby on developer community.

[JuanZamudioGBM]

Can we have a tree view based on the namespaces? right now the hierarchy view is not very useful.

Right now its something like this:

TestProject
    Company.FeatureA.SubFeatureA(3)
        TestClassA(1)
            MethodName1(1)
        TestClassB(1)
            MethodName2(1)
        TestClassC(1)
            MethodName3(1)
    Company.FeatureA.SubFeatureB(1)
        TestClassD(1)
            MethodName4(1)
    Company.FeatureB.SubFeatureC(5)
        TestClassE(1)
           MethodName5(1)
        TestClassF(1)
           MethodName6(1)
        TestClassG(1)
           MethodName7(1)

I think it will be more useful something like this:

TestProject
    Company
        FeatureA
            SubFeatureA(3)
                TestClassA(1)
                    MethodName1(1)
                TestClassB(1)
                    MethodName2(1)
                TestClassC(1)
                    MethodName3(1)
            SubFeatureB(1)
                TestClassD(1)
                    MethodName4(1)
            SubFeatureC(5)
                TestClassE(1)
                    MethodName5(1)
                TestClassF(1)
                    MethodName6(1)
                TestClassG(1)
                    MethodName7(1)

[jerluo]

Please bring back sorting all tests in alphabetic order.

[fergusonr]

I'd love to have the ability to create your own sub-hierarchy within a namespace / class driven by the user overridden test name

[TestCase(@"data\rollleft.mdl", TestName = @"Movement\Roll left")]
[TestCase(@"data\rollright.mdl", TestName = @"Movement\Roll right")]
[TestCase(@"data\walkforward.mdl", TestName = @"Movement\Walk forward")]
public void Runner(string testdata)
{

[zaus]

I arrived here from the vs 2019 docs article suggestion which is dated 6/25/2020 but still shows the 2017 hierarchy. I would really like the return of the "old" configurable hierarchy grouping, namely being able to see ALL tests sorted alphabetically and grouped by outcome. The current forced hierarchy by class is practically unusable when years of (inherited) unit tests have relied on alphabetical and/or text-segment filtering for organization.

I don't need an additional column indicated "State" when there's already an icon much more clearly and concisely indicating it; my laptop screen real-estate is valuable. Nor do I need the additional padding from multi-level trees which only serves to further reduce the usable space.

[Evangelink]

Ping @AbhitejJohn in case you want to do something from this feedback loop.

[y87feng]

For sorting by outcome, there is a workaround. You can right click on the column header, open the context menu, and then select "State". Then, clicking the column header "State" will enable sorting

[Evangelink]

@AbhitejJohn I am closing this ticket. Ping me if you want to keep it open to track any work on your side.

[kvart714]

Hi @Evangelink
I'd like to see the tests in the order they appear in the code.
I didn't find this option