Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running Parallel Tests in Suite #187

Open
eyphka opened this issue Jun 23, 2015 · 12 comments
Open

Running Parallel Tests in Suite #187

eyphka opened this issue Jun 23, 2015 · 12 comments

Comments

@eyphka
Copy link

@eyphka eyphka commented Jun 23, 2015

When running multiple tests in a suite which have been set to suite.t().Parallel(), tests will pass even if they should fail.

@dwlnetnl
Copy link

@dwlnetnl dwlnetnl commented Sep 22, 2015

That's true, but as it's currently structured it isn't possible to run suite tests in parallel. This is because before and after each suite test SetupTest and TearDownTest is run.

@ernesto-jimenez
Copy link
Member

@ernesto-jimenez ernesto-jimenez commented Nov 2, 2015

@eyphka do you want to make a PR to address it? :)

@ernesto-jimenez
Copy link
Member

@ernesto-jimenez ernesto-jimenez commented Nov 2, 2015

Related: #52

@eyphka
Copy link
Author

@eyphka eyphka commented Nov 3, 2015

Sure

On Sunday, November 1, 2015, Ernesto Jiménez notifications@github.com
wrote:

Related: #52 #52


Reply to this email directly or view it on GitHub
#187 (comment).

@oryband
Copy link

@oryband oryband commented May 8, 2016

Reviving this issue. Presumably it should be possible to run suite tests in parallel.

The problem with parallelism here is that the suite members used in tests have a shared single instance for all. Tests running in parallel could potentially access these members at the same time, causing them to "collide" with each other via these members.

However, my suggestion is that testify will add a special Parallel() flag to the suite. This new flag will instantiate new instance members for every new Parallel() test, calling Setup/TearDownTest() as well.

This will of course require extra work from the test developer, to make sure Setup/TearDownTest() are suitable for being called with/without parallel (As some tests will be run in parallel, and some don't).

Thoughts?

@dansimau
Copy link

@dansimau dansimau commented Nov 20, 2016

I just whipped up a slightly different approach to running tests in parallel:
https://gist.github.com/dansimau/128826e692d7834eb594bb7fd41d2926

Similar to #369, it introduces a RunParallel, except that it uses reflection to create a new instance of the suite for each test run.

Because a new suite is initialised before every test, it means tests can't share data on the suite struct. This is a change in behaviour, though I imagine you could change the patch to make a copy of the suite struct instead of initialising a new one (that way tests could still share data and setup/teardown code using SetupSuite).

I actually prefer the share-nothing approach though; it feels like a bad practice to modify data on the suite struct while tests are running, especially now in parallel. I can't see why tests should ever share data (outside what is set up in SetupTest); but maybe I'm missing a use case.

@varunrau
Copy link

@varunrau varunrau commented Oct 18, 2017

is it still the case that suite tests can't be run in parallel?

@alessio
Copy link

@alessio alessio commented Sep 30, 2020

Hi! Any updates on this? Are you planning to do work on this feature request?

@ysmood
Copy link

@ysmood ysmood commented Oct 1, 2020

One solution:

package main_test

import (
	"testing"
	"time"

	"github.com/stretchr/testify/assert"
	"github.com/ysmood/got"
)

func Test(t *testing.T) {
	got.Each(t, beforeEach)
}

func beforeEach(t *testing.T) Suite {
	t.Parallel()
	return Suite{assert.New(t)}
}

type Suite struct { // struct that holds subtests
	*assert.Assertions
}

func (s Suite) A() { // test case A
	time.Sleep(time.Second)
	s.Equal(1, 1)
}

func (s Suite) B() { // test case B
	time.Sleep(time.Second)
	s.Equal(2, 1)
}

As you can see, test A is not affected:

$ go test       
--- FAIL: Test (0.00s)
    --- FAIL: Test/B (1.00s)
        main_test.go:31: 
            	Error Trace:	main_test.go:31
            	            				value.go:475
            	            				value.go:336
            	            				each.go:42
            	            				value.go:564
            	            				asm_amd64.s:20
            	Error:      	Not equal: 
            	            	expected: 2
            	            	actual  : 1
            	Test:       	Test/B
FAIL
exit status 1
FAIL	goplay/default	1.254s
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants