Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Container orchestration refactor #257

Closed
wpbonelli opened this issue Jan 15, 2022 · 2 comments
Closed

Container orchestration refactor #257

wpbonelli opened this issue Jan 15, 2022 · 2 comments
Labels
in progress Someone is working on this issue (please also assign yourself) priority Should be resolved first, if possible python Pull requests that update Python code

Comments

@wpbonelli
Copy link
Member

wpbonelli commented Jan 15, 2022

There's little justification to roll our own system on top of Dask to interface with deployment targets. There's no lack of options in this space, and Dask is better suited to interactive use cases anyway, e.g., in Jupyter notebooks. (It would be nice to provide preconfigured notebooks as tutorials alongside workflows, with examples for how to use the code and parallelize it with Dask, but that's a separate concern.)

Snakemake or NextFlow do everything we need and more:

  • running jobs in containers, including support for Singularity
  • integration with cluster schedulers
  • environment variables and parametrization

The plantit-cli should just expose a handy interface to the plantit resource APIs. (Maybe it's worth spinning off the Terrain interface as its own project -- it might be nice to have a CLI to Terrain, although it should probably disallow data transfers beyond a certain size limit to avoid overloading CyVerse web APIs)

@wpbonelli wpbonelli added python Pull requests that update Python code priority Should be resolved first, if possible labels Jan 15, 2022
@wpbonelli wpbonelli added the in progress Someone is working on this issue (please also assign yourself) label Jan 16, 2022
wpbonelli added a commit that referenced this issue Jan 18, 2022
@wpbonelli
Copy link
Member Author

This is a good opportunity to introduce task composition: check input/output types of each workflow and, if compatible, allow chaining. We should also allow task groups to be scheduled in parallel.

wpbonelli added a commit that referenced this issue Feb 15, 2022
wpbonelli added a commit that referenced this issue Feb 15, 2022
wpbonelli added a commit that referenced this issue Feb 16, 2022
wpbonelli added a commit that referenced this issue Feb 16, 2022
wpbonelli added a commit that referenced this issue Feb 16, 2022
wpbonelli added a commit that referenced this issue Feb 17, 2022
wpbonelli added a commit that referenced this issue Feb 18, 2022
wpbonelli added a commit that referenced this issue Feb 18, 2022
wpbonelli added a commit that referenced this issue Feb 19, 2022
wpbonelli added a commit that referenced this issue Feb 20, 2022
@wpbonelli
Copy link
Member Author

Tentatively finished with 36ffc8d

wpbonelli added a commit that referenced this issue Feb 21, 2022
wpbonelli added a commit that referenced this issue Feb 21, 2022
wpbonelli added a commit that referenced this issue Feb 21, 2022
wpbonelli added a commit that referenced this issue Feb 21, 2022
wpbonelli added a commit that referenced this issue Feb 21, 2022
wpbonelli added a commit that referenced this issue Feb 21, 2022
wpbonelli added a commit that referenced this issue Feb 21, 2022
wpbonelli added a commit that referenced this issue Feb 21, 2022
wpbonelli added a commit that referenced this issue Feb 22, 2022
wpbonelli added a commit that referenced this issue Feb 22, 2022
wpbonelli added a commit that referenced this issue Feb 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
in progress Someone is working on this issue (please also assign yourself) priority Should be resolved first, if possible python Pull requests that update Python code
Projects
None yet
Development

No branches or pull requests

1 participant