Modular Workflow Patterns

Notebook on the development of a modular workflow pattern for RepeatMasker

Goals

Support a wide variety of cluster/cloud computing environments for the parallel steps of RepeatMasker. These are primarily in the areas of sequence alignment and alignment adjudication.
Minimize dependencies and installation complexity for users.
Provide a user-friendly command-line interface (CLI) and logging facilities.
Support the user of RepeatMasker as a module in a more complex analysis pipeline.
Store/maintain the provenance of workflow runs to support reproducibility and incremental updates.
Facilitate user-support by keeping the workflow logic/tools as simple and transparent as possible.

The workflow orchestration ecosystem is still rapidly evolving. I expect new options to emerge that may be better than the current players.
The current dominant players are Nextflow and Snakemake, with the former having an edge in terms of support, and use at large computing centers. I don't like either option to be honest. They are complex, and difficult to support.
It would be best if we could develop a pattern that is likely to work with current and future workflows, and implement it using the best available manager.
The new RepeatMasker should modular enough to allow orchestration by a variety of workflow managers. This modularity will likely be in the form of Python scripts that can be called from the command line, and that use the file system for input/output.