[suite] Should testify.Suite run all tests from the sub suite in the case when passed -testify.m flag?

[denis-tingaikin]

Hello guys,

We've found one funny thing about suite.Run. It allows running inner suites (that look awesome to me :) ). But this working some strange if passed testify.m parameter.

For example:

package mypkg_test

import "github.com/stretchr/testify/suite"

type suite1 struct{
    suite.Suite
}

func (s *suite1) Test1() {
    println("test1")
}

type suite2 struct{
    suite.Suite
}

func (s *suite2) Test2() {
    suite.Run(s.T(), new(suite1))
}

func TestSuite2(t *testing.T) {
    suite.Run(t, new(suite2))
}

So if I just run TestSuite2 then it works fine (suite2 runs all tests from suite1).
But if I run TestSuite2 with paramter -testify.m="Test2" then suite1 will be ignored.

Please share your thoughts about:

1. Is sub suites legal?
2. Should the matching filter ignore sub suites?

[denis-tingaikin]

Potential fix: #1051

[boyan-soubachov]

Looking at the description of the testify.m parameter, it's meant to match tests rather than suites. The behaviour that you have noticed therefore seems correct to me.

Also, changing this would necessitate a breaking change and therefore would be merged in v2 at the earliest.

[denis-tingaikin]

@boyan-soubachov OK,
Do you have an idea about a pure fix for the issue?

I see these solutions:

1. Add a custom filter option as it has done in feat: allow passing custom test filter function for suites #1051.
2. Add an additional flag that will represent something like suite name regex. If the flag is passed then the flag with test matching will be applying only for the passed suite. For example: go test ./... -testify.m="Test1" -testify.s="Suite1" will mean something like: run all tests but for suite1 run only Test1.
3. Apply to testify.m flag only for top-level suites
4. Add an alternative flag that will expect a go-template string. With go-templates testify could support more interesting filters.
5. Your solution :)