Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallelize training #100

Closed
thcrock opened this issue Apr 11, 2017 · 0 comments
Closed

Parallelize training #100

thcrock opened this issue Apr 11, 2017 · 0 comments
Assignees
@thcrock
Labels
performance

Comments

@thcrock
Copy link
Contributor

thcrock commented Apr 11, 2017

This is the one time-consuming that isn't parallelized in LocalParallelPipeline yet.
@thcrock thcrock self-assigned this Apr 11, 2017
thcrock added a commit that referenced this issue Apr 11, 2017
@thcrock
Parallelize Training [Resolves #100]
f25f51a 
- Restructure ModelTrainer to make tasks using generate_train_tasks, and process them using process_train_task. train_models and generate_trained_models still exist but use this internally.
- Have LocalParallelPipeline use new ModelTrainer interface to parallelize training
- Remove deprecated matrix_store arg in ModelTrainer constructor, all references and test
- Add assertion to prevent InMemoryModelStorageEngine from being used with LocalParallelPipeline to prevent need for shared memory management between trainer processes (could be added in future if desired), and change pipeline test to use FSModelStorageEngine
- Have FSStore make directory structure if it doesn't exist
- Move ModelTrainer#replace to constructor
ecsalomon added a commit that referenced this issue Apr 18, 2017
@ecsalomon
Merge pull request #102 from dssg/parallelize_training
dce910a 
Parallelize Training [Resolves #100]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@thcrock thcrock

Labels
performance
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@thcrock