Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support parallelism #28

Closed
thcrock opened this issue Mar 1, 2017 · 0 comments
Closed

Support parallelism #28

thcrock opened this issue Mar 1, 2017 · 0 comments
Assignees
@thcrock
Milestone
v0.3

Comments

@thcrock
Copy link
Contributor

thcrock commented Mar 1, 2017 •
edited
Loading

Although it is possible to support parallelism of triage components by using specific triage components instead of the whole pipeline, it should also support some simple version of this out of the box.

An injected executor could be a good way to do this (ie SimpleExecutor, MultiprocessingExecutor).

A better first pass, now that pipeline.run() is more compact, would be to subclass Pipeline and only reimplement .run().
@thcrock thcrock added this to the v0.3 milestone Mar 22, 2017
@thcrock thcrock self-assigned this Mar 22, 2017
thcrock added a commit that referenced this issue Mar 23, 2017
@thcrock
Support parallelism [Resolves #28]
e92f733 
- Create abstract class PipelineBase, which sets up components and requires subclasses to implement run
- Move current Pipeline to SerialPipeline, subclass of PipelineBase
- Create LocalParallelPipeline subclass of PipelineBase to spawn workers using concurrent.futures.ProcessPoolExecutor. Currently just parallelizing the testing phase
- Make poolclass configurable, as NullPool was giving some problems with multiple processes and QueuePool seems to be more robust
- Modify MettaCSVMatrixStorage to only load data upon use, improving serializability
ecsalomon added a commit that referenced this issue Mar 24, 2017
@ecsalomon
Merge pull request #68 from dssg/parallelize
8def9b2 
Support parallelism [Resolves #28]
jesteria pushed a commit that referenced this issue Nov 16, 2017
@thcrock
Fix replace bug
9c08327 
- Fix logging argument error in IndividualImportance (it doesn't raise exception, but does show up as an error in the log)
- Fix replace logic [Resolves #28]
jesteria pushed a commit that referenced this issue Nov 28, 2017
@mbauman
Merge pull request #28 from dssg/ep/format_kwargs
b46832e 
enable quantities to take formatting kwargs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@thcrock thcrock

Labels
None yet
Projects
None yet
Milestone
v0.3
Development

No branches or pull requests
1 participant
@thcrock