Run specs in one suite with different label

[liubog2008]

For example:

func TestE2E(t *testing.T) {
	gomega.RegisterFailHandler(ginkgo.Fail)
	features := []string{
		"a",
		"b",
		"c",
	}
	for _, f := range features {
		t.Run("feature="+f, func(t *testing.T) {
			ginkgo.RunSpecs(t, "E2E with feature "+f, ginkgo.Label(f))
		})
	}
}

Why not use multiple suites?

Because features maybe too many

Why not call ginkgo multiple times?

I hope to get a summary report.

Why not run once with all features?

1. Some features cannot run with other features at same time.
2. BeforeSuite is too expensive.
3. All cases with one feature can be run in parallel.

[onsi]

Hey there @liubog2008 - I think I understand what you're trying to do and why. I don't currently have a strong answer for you. It seems that a common use-case in Kubernetes is:

1. Use (Synchronized)BeforeSuite to do some very expensive set up.
2. Have families of tests that each need their own less expensive (but still fairly expensive - so it would be prohibitive to do in a BeforeEach) amount of shared set up.
3. Sometimes multiple "features" can run in parallel. Sometimes they must run separately. Sometimes the tests for a given feature can be run in parallel. Sometimes they must run separately. It's fairly complex.

Ginkgo doesn't have a strong answer for this matrix of use-cases... yet. It's currently designed around the notion that:

1. The initial (Synchronized)BeforeSuite is very expensive.
2. All individual specs are independent and so have their own setup and teardown and can be scheduled in parallel with each other.

Over the years we've gradually added some additional options:

• The Serial decorator allows you to specify that a given spec cannot run in parallel with any other spec.
• The Ordered decorator allows you to specify that a group of specs have a shared setup (BeforeAll) and must run on the same parallel process (so they run in serial with-respect-to each other, but can be in parallel with other specs).

I'm considering options to allow for something more general so you can express things like "this group of tests can run in parallel with respect to each other but not with respect to other tests" but I only have some ideas at this point, not a concrete plan or proposal yet.

Until then, fwiw:

Why not call ginkgo multiple times?
I hope to get a summary report.

You can get this with a little bit of code that loads and merges multiple JSON/JUnit reports.

[liubog2008]

Yes, I'm worked in Kubernetes. The biggest problem is that I have to do something between two subset of tests.

some tests -> do something serial and expensive(e.g. upgrade some apps and the upgrade also should be tested) -> other tests

[onsi]

Thanks for the detail - it's helpful and further paints a picture of just how much flexibility folks seem to need when building these complex e2e suites.

I have a few more questions as I'm actively considering how to design solutions for this kind of problem space:

• When the "serial and expensive" thing happens are other (unrelated) tests allowed to run in parallel? For example, say you have a set of tests related to Feature A and a set of tests related to Feature B. Feature A looks like "some tests -> do something serial and expensive -> other tests" and Feature B also looks like "some tests -> do something serial and expensive -> other test". Are both Feature A and Feature B allowed to run in parallel with each other (i.e. on different Ginkgo parallel processes)? Or does "something serial and expensive" really need to be serial across the entire cluster (i.e. Feature B would have to wait before proceeding if Feature A is doing its serial and expensive thing)?

• How do you currently run in parallel? Do you have a single cluster and each test has its own namespace and so when running in parallel there is no interference?

• Assuming Feature A and Feature B can run at the same time then what you are asking for can be accomplished with a single massive It for each of A and B. Within the It you could manage some goroutines and aggregate failures yourself. Obviously this isn't elegant and leads to less effective test reporting - but I'd love to understand the issues you see with approaching things that way so I can better understand how to design a more general solution.

• Have you actually tried either of these options:

/*
option 1: this runs "some tests" in parallel, then does the upgrade once and runs "other tests" in serial
*/

Describe("Feature A", func() {
    BeforeEach(func() {
        // set up the application
    }, OncePerOrdered)

    // some tests
    It("tests something", func() { ... })
    It("tests something else", func() { ... })
  
    // do something serial and expensive
    Describe("When upgraded", Ordered, func() {
        BeforeAll(func() {
            // the expensive upgrade
        })

        // other tests
        It("tests something after the upgrade", func() { ... })
        It("tests another thing after the upgrade", func() { ... })
    }) 
})


/*
option 2: this runs everything in parallel.  It's true that the upgrade is very expensive - but with enough parallelism and resources the actual run-time is not too bad.
*/

Describe("Feature A", func() {
    BeforeEach(func() {
        // set up the application
    }, OncePerOrdered)

    // some tests
    It("tests something", func() { ... })
    It("tests something else", func() { ... })
  
    // do something expensive, repeatedly, for each test
    Describe("When upgraded", func() {
        BeforeEach(func() {
            // the expensive upgrade
        })

        // other tests
        It("tests something after the upgrade", func() { ... })
        It("tests another thing after the upgrade", func() { ... })
    }) 
})

and evaluated their performance characteristics when running in parallel with ginkgo -p? I understand that the operation in question is expensive. When I've been faced with situations like this I've either (a) used a single large It documented with Bys, (b) simply paid the cost of running the upgrade for each spec but running in parallel which amortizes that cost (option 2), (c) used an Ordered container with a BeforeAll as shown above (option 1).