Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow parallel execution of envs (merge or replace detox?) #439

Closed
ssbarnea opened this issue Jan 17, 2017 · 15 comments · Fixed by #1102
Closed

Allow parallel execution of envs (merge or replace detox?) #439

ssbarnea opened this issue Jan 17, 2017 · 15 comments · Fixed by #1102

Comments

@ssbarnea
Copy link
Member

@ssbarnea ssbarnea commented Jan 17, 2017

This is more of an idea that would simplify parallel execution of tests using tox. Why not merging detox into tox?

I do see several benefits from this:

  • better visibility
  • simplify project management, they already share the same issue tracker
  • easier to deploy
  • more trust: detox project has last update from 2015 on pypi making people believe it may be abandonware, also the document page contain outdated links to bitbucket.
  • once done, pypi admins can replace detox project with an alias to tox (they already did this for other projects).

As part of the proposed consolidation we can even trigger the parallel option via -P command line option.

@hpk42
Copy link
Contributor

@hpk42 hpk42 commented Jan 17, 2017

could you look into replacing the usage of eventlet/greenlets with plain threads with detox? That's the main block from my point of view for integrating detox into tox as it would introduce a new maybe not fully stable dependency (not sure what the state of eventlet currently is and if it can be installed everywhere etc.).

@ssbarnea
Copy link
Member Author

@ssbarnea ssbarnea commented Jan 17, 2017

@hpk42 Thanks for the quick reply. Glad to hear that you don't have anything against it. I will have a look.

@davidism
Copy link

@davidism davidism commented Jan 19, 2017

If this happens, I'd also like to be able to mark some envs as "synchronous" so that, for example, coverage collection only happens after all the parallel tests run.

@obestwalter obestwalter changed the title merge detox into tox detox: merge detox into tox Feb 21, 2017
@obestwalter
Copy link
Member

@obestwalter obestwalter commented Feb 21, 2017

I just saw in the README that detox only runs with 2.6 and 2.7 is this still true? This would have to be considered when merging also, depending on what the reason fro this is.

@pawamoy
Copy link

@pawamoy pawamoy commented Mar 1, 2017

Well detox is now tested against 2.7, 3.4, 3.5 and 3.6: https://travis-ci.org/tox-dev/detox, so maybe we could remove the note in detox's README.

@davidism
Copy link

@davidism davidism commented Mar 1, 2017

I've installed and used it from Python 3 envs without issue.

@obestwalter
Copy link
Member

@obestwalter obestwalter commented Mar 6, 2017

Hi @hpk42,

README says:

detox runs only on python2.6 and python2.7 (but supports creation of python3 and all environments supported of the underlying "tox" command)

is this still valid?

@nirs
Copy link

@nirs nirs commented Jul 2, 2017

Running multiple processes should be easy without eventlet, just little more code.
Is it ok to depend on concurrent.futures, or we prefer to have no dependencies?

@ssbarnea
Copy link
Member Author

@ssbarnea ssbarnea commented Jul 2, 2017

I vote for using dependencies, if they are reliable and maintained, like those that are going to become part of future version of python.

detox works with most versions of python, the info is outdated.

@jfboismenu
Copy link

@jfboismenu jfboismenu commented Nov 4, 2017

Has anyone started work on this? I’m open to taking a stab at it. I’d love to have everyone at work use only tox and add support for environment dependency management so collection jobs (like collecting coverage from different builds into a single report) could be run after a subset of environments have been executed.

@obestwalter
Copy link
Member

@obestwalter obestwalter commented Nov 5, 2017

Hi @jfboismenu - thanks, have a go. I'd love to have this merged and as far as I know, nobody is working on this.

@ssbarnea
Copy link
Member Author

@ssbarnea ssbarnea commented Nov 6, 2017

Yeah, feel free to takeover this. Sadly I am too busy with other bugs to have time to implement this but if you raise a PR, I could help reviewing it. Thanks.

@gaborbernat
Copy link
Member

@gaborbernat gaborbernat commented Sep 8, 2018

The way I see detox will remain separate. That being said tox can/should get a parallel flag; however the implementation of this would have little to do with the current detox; instead would be an alternative. Will tackle this after having pep-517/8 support and pyproject.toml configuration; as the order of importance.

@gaborbernat gaborbernat self-assigned this Sep 8, 2018
@gaborbernat gaborbernat changed the title detox: merge detox into tox allow parallel execution of envs Sep 8, 2018
@gaborbernat gaborbernat added this to the 3.5 milestone Sep 18, 2018
@obestwalter obestwalter changed the title allow parallel execution of envs Allow parallel execution of envs (merge or replace detox?) Sep 22, 2018
@gaborbernat gaborbernat removed this from the 3.5 milestone Oct 8, 2018
@gaborbernat gaborbernat added this to the 3.6 milestone Oct 8, 2018
@gaborbernat gaborbernat removed this from the 3.6 milestone Oct 8, 2018
@gaborbernat gaborbernat added this to the 3.6 milestone Oct 8, 2018
@obestwalter
Copy link
Member

@obestwalter obestwalter commented Nov 2, 2018

Hi @gaborbernat, what would be different with the parallel flag?

I would really like to see detox vanish into tox core, because as a free standing project it is de facto unmaintained. So when tox grows this parallel functionality I would propose that we mark detox as obsolete and unmaintained. See https://github.com/tox-dev/detox/issues/27

@gaborbernat
Copy link
Member

@gaborbernat gaborbernat commented Dec 16, 2018

A minimal viable implementation is now available under #1102, anyone interested feel free to review and submit your thoughts.

gaborbernat added a commit to gaborbernat/tox that referenced this issue Dec 16, 2018
gaborbernat added a commit to gaborbernat/tox that referenced this issue Dec 16, 2018
gaborbernat added a commit to gaborbernat/tox that referenced this issue Dec 16, 2018
@gaborbernat gaborbernat removed this from the 3.6 milestone Dec 16, 2018
@gaborbernat gaborbernat added this to the 3.7 milestone Dec 16, 2018
gaborbernat added a commit to gaborbernat/tox that referenced this issue Jan 9, 2019
gaborbernat added a commit to gaborbernat/tox that referenced this issue Jan 10, 2019
gaborbernat added a commit that referenced this issue Jan 11, 2019
This solution is ``subprocess`` based. We're already heavily using subprocesses, so I consider this implementation safe. 

Resolves #439.
@helpr helpr bot added pr-merged and removed pr-available labels Jan 11, 2019
@tox-dev tox-dev locked and limited conversation to collaborators Jan 14, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

8 participants