New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running Parallel Tests in Suite #187

Open
eyphka opened this Issue Jun 23, 2015 · 10 comments

Comments

Projects
None yet
7 participants
@eyphka
Copy link

eyphka commented Jun 23, 2015

When running multiple tests in a suite which have been set to suite.t().Parallel(), tests will pass even if they should fail.

@dwlnetnl

This comment has been minimized.

Copy link

dwlnetnl commented Sep 22, 2015

That's true, but as it's currently structured it isn't possible to run suite tests in parallel. This is because before and after each suite test SetupTest and TearDownTest is run.

@ernesto-jimenez

This comment has been minimized.

Copy link
Member

ernesto-jimenez commented Nov 2, 2015

@eyphka do you want to make a PR to address it? :)

@ernesto-jimenez

This comment has been minimized.

Copy link
Member

ernesto-jimenez commented Nov 2, 2015

Related: #52

@eyphka

This comment has been minimized.

Copy link

eyphka commented Nov 3, 2015

Sure

On Sunday, November 1, 2015, Ernesto Jiménez notifications@github.com
wrote:

Related: #52 #52


Reply to this email directly or view it on GitHub
#187 (comment).

@oryband

This comment has been minimized.

Copy link

oryband commented May 8, 2016

Reviving this issue. Presumably it should be possible to run suite tests in parallel.

The problem with parallelism here is that the suite members used in tests have a shared single instance for all. Tests running in parallel could potentially access these members at the same time, causing them to "collide" with each other via these members.

However, my suggestion is that testify will add a special Parallel() flag to the suite. This new flag will instantiate new instance members for every new Parallel() test, calling Setup/TearDownTest() as well.

This will of course require extra work from the test developer, to make sure Setup/TearDownTest() are suitable for being called with/without parallel (As some tests will be run in parallel, and some don't).

Thoughts?

@andrew-plunk

This comment has been minimized.

Copy link

andrew-plunk commented Nov 10, 2016

I have a test suite runner here that supports testing.T.Parallel().
andrew-plunk@d3e6d55

The problem with supporting parallel execution on Suite is that the TestingSuite's public interface T() returns a *testing.T. It is very hard to make this goroutine safe. Rather than wrapping access to the returned *testing.T, I went down the path of removing the dependencies on the Suite struct's internal state, and simply pass the internal tests' *testing.T in via function arguments.

This removes the ability to call Assert() on the testing suite, but is a minior inconvenience IMO because the assert package supports passing a *testing.T object into it's functions.

Let me know how you all feel about this strategy. I will work on cleaning up that branch and adding docs/tests for a pr if it works for you.

@dansimau

This comment has been minimized.

Copy link

dansimau commented Nov 20, 2016

I just whipped up a slightly different approach to running tests in parallel:
https://gist.github.com/dansimau/128826e692d7834eb594bb7fd41d2926

Similar to #369, it introduces a RunParallel, except that it uses reflection to create a new instance of the suite for each test run.

Because a new suite is initialised before every test, it means tests can't share data on the suite struct. This is a change in behaviour, though I imagine you could change the patch to make a copy of the suite struct instead of initialising a new one (that way tests could still share data and setup/teardown code using SetupSuite).

I actually prefer the share-nothing approach though; it feels like a bad practice to modify data on the suite struct while tests are running, especially now in parallel. I can't see why tests should ever share data (outside what is set up in SetupTest); but maybe I'm missing a use case.

@andrew-plunk

This comment has been minimized.

Copy link

andrew-plunk commented Nov 23, 2016

@dansimau I think your approach is closer to the current architecture of testify's Suite.
I think there may be a problem with the way you copy the suite. You need to do a deepcopy rather than just create an instance of that type. https://play.golang.org/p/b83RgXGOQJ

Otherwise the work done in the SetupSuite function calls will be lost.

@andrew-plunk

This comment has been minimized.

Copy link

andrew-plunk commented Nov 23, 2016

@dansimau I updated #369 to use the deepcopy strategy referenced above.

@varunrau

This comment has been minimized.

Copy link

varunrau commented Oct 18, 2017

is it still the case that suite tests can't be run in parallel?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment