Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge python-pipeline into this project #9

Open
MikeDacre opened this issue Jun 28, 2016 · 1 comment
Open

Merge python-pipeline into this project #9

MikeDacre opened this issue Jun 28, 2016 · 1 comment
Assignees
Milestone

Comments

@MikeDacre
Copy link
Owner

The python-pipeline project is an effort to make it easy to create complex pipelines with python. It isn't that useful outside of a multithreading environment, so it makes sense to merge it in here and implement native multithreading in that project through the cluster module.

Rather than keep it as a separate project, the pipeline package should be added as a separate package alongside cluster to be used if the user wishes. However, it is important that its usage is not required in order to use the cluster package. ie. pipeline should depend on cluster, but cluster should not depend on pipeline.

@MikeDacre MikeDacre added this to the 0.8.0 milestone Jun 28, 2016
@MikeDacre MikeDacre self-assigned this Jun 28, 2016
@MikeDacre MikeDacre modified the milestones: 0.9.0, 0.8.0 Oct 27, 2016
@MikeDacre
Copy link
Owner Author

Most of the pipeline code is not needed, but add a Pipeline class that jobs can be added to.

This class should write itself to disk and be easily restorable from the name and directory alone. All jobs will have the track option in #80 set, which will allow pipeline restoration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant