New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Force test suites to run serially instead of concurrently WAS: separate configuration for test execution #492

Closed
gregwebs opened this Issue Jul 2, 2015 · 12 comments

Comments

Projects
None yet
4 participants
@gregwebs
Contributor

gregwebs commented Jul 2, 2015

Our example use case is that we always want build parallelism (including building of the test suite).
However, we would like to run the test suites serially (configure as jobs: 1) to avoid contending over shared resources such as the database.

@snoyberg

This comment has been minimized.

Contributor

snoyberg commented Jul 2, 2015

I'm concerned about the implied scope of this based on the title: "separate configuration" can mean a lot of things, whereas "don't run tests in parallel" means only one thing. Is there some other example of configuration you want control over like this?

@gregwebs

This comment has been minimized.

Contributor

gregwebs commented Jul 2, 2015

Right now I don't have other use cases. One thing I can think of that I have needed before (but probably shouldn't because it is really a hack) is setting flags differently.

@gregwebs

This comment has been minimized.

Contributor

gregwebs commented Jul 2, 2015

Original issues was #482. @pseudonom @mboes

@mboes

This comment has been minimized.

Contributor

mboes commented Jul 3, 2015

The configuration provides a default for all commands. Which can be overridden on the command line for specific commands. While it may be useful to provide per-command defaults in the config, could stack test --jobs=1 do the job for now assuming that forcibly serializing test suite execution is something you view as a temporary hack? Or is pinning jobs: 1 for tests something you want the ability to do forever for some reason?

@gregwebs

This comment has been minimized.

Contributor

gregwebs commented Jul 3, 2015

The main problem with that solution is that it serializes the build process but we only want to serialize the test execution.

@snoyberg

This comment has been minimized.

Contributor

snoyberg commented Jul 3, 2015

@gregwebs You don't have jetlagged kids, why are you awake right now?!?

I'm in favor of having something like concurrent-tests: BOOL in the YAML file. I doubt this will be the only report we get of test suites that share some resource. I'm actually in the middle of implementing this right now.

@snoyberg snoyberg changed the title from separate configuration for test execution to Force test suites to run serially instead of concurrently WAS: separate configuration for test execution Jul 3, 2015

snoyberg added a commit that referenced this issue Jul 3, 2015

@snoyberg

This comment has been minimized.

Contributor

snoyberg commented Jul 3, 2015

OK, I've pushed a commit to recognize the concurrent-tests setting in stack.yaml. @gregwebs @pseudonom can you test this and let me know if it resolves the issue? If so, would you be able to update the stack.yaml Wiki page with a description of the feature?

@snoyberg snoyberg added this to the 0.2.0.0 milestone Jul 3, 2015

@pseudonom

This comment has been minimized.

pseudonom commented Jul 3, 2015

I can test this on Monday. I'll update the wiki when I do so.

@gregwebs

This comment has been minimized.

Contributor

gregwebs commented Jul 6, 2015

I tested the concurrent-tests setting. It both builds the projects in serial and runs the tests in serial. This is an improvement for us because we can still build in parallel with stack build. We can run stack build && stack test and that is closer to what we want. Even still, the test suites themselves are not built in parallel.

The desired behavior is only to run the tests in serial, but for all building (including of the test suites) to happen in parallel.

@gregwebs

This comment has been minimized.

Contributor

gregwebs commented Jul 6, 2015

I tested with a stack version that had the concurrent-tests setting commit: Version 0.1.1.0, Git revision 38fa3cf

It appears that we can also test with the 0.2 release now.

@snoyberg

This comment has been minimized.

Contributor

snoyberg commented Jul 6, 2015

Hmm, that's surprising. I'll wait to hear on the results of trying this
with 0.2 before investigating further.

On Sun, Jul 5, 2015 at 6:11 PM Greg Weber notifications@github.com wrote:

I tested with a stack version that had the concurrent-tests setting
commit: Version 0.1.1.0, Git revision 38fa3cf
38fa3cf

It appears that we can also test with the 0.2 release now.


Reply to this email directly or view it on GitHub
#492 (comment)
.

@snoyberg

This comment has been minimized.

Contributor

snoyberg commented Jul 13, 2015

Last I heard this was working as expected, so closing. If there are still issues, please reopen.

@snoyberg snoyberg closed this Jul 13, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment