Smoothing the config overhead of workflows

[AndrewSkelton]

Hi,

Thanks for the great work on the Salmon / Alevin eco system :)

I've been implementing various single cell workflows with containerisation, and have been testing out CITE seq data with CR and SimpleAF. The CR execution is fairly straightforward once the libraries csv file is created, and was hoping it'd be an analogous mechanism with SimpleAF. I've checked out workflows and it's great that there's already some common templates in there! The sticking point I find is the manual intervention of filling out the workflow json very difficult to automate (perhaps I've missed a step which has this case in mind).

I understand these processes can be quite heavy parameter wise, but command line accessors to elements of the workflow template might ease this burden? If this flexibility doesn't exist, I guess the best port of call would be to chain the commands manually?

Thanks,

Andrew

[rob-p]

Hi @AndrewSkelton,

Thanks for raising this discussion. I agree that this currently a difficulty and it's something that we haven't yet solved from the design perspective, but which both @DongzeHE and I are eager to build a solution for.

In short, the current workflow process (the workflow workflow, if you will ;P) works well to run a complicated workflow that could involve many simpleaf (and even external) commands once. That is, it makes it easy to run a CITE-seq experiment. However, you're right that when you have many distinct samples that you want to process with the same workflow, there's a lot of overhead.

Essentially, what one would like is the ability to take input from e.g. a CSV file or some other similarly structured file, and "patch" it into a workflow. That is to create many instantiated templates from it (i.e. things that can actually be executed). While we currently don't have this implemented, we'd definitely like to have your suggestion on what the easiest interface for something like this would be that would nonetheless still be flexible for many different types of workflows. This is the kind of things that jasonnet (and nickel that we are looking at as a possible replacement) certainly support, and a feature that we'd like to have, but which we haven't figured out the proper design for yet.

Thanks,
Rob

[DongzeHE]

I think the most straightforward solution for now is to have a Nextflow workflow wrapping simpleaf workflow in. I can write one during the weekend so that @AndrewSkelton can use it.

-Dongze

[DongzeHE]

Hi @rob-p and @AndrewSkelton ,

I made a nextflow workflow https://github.com/DongzeHE/simpleaf_citeseq_nextflow for processing CITE-seq using simpleaf. This workflow accepts a sample sheet, so you can analyze as many samples as you want in one run and build the index only once.

This workflow fills up the simpleaf template for CITE-seq under the hood, invokes a simpleaf index command separately to generate an index folder, and uses the generated index everywhere in the workflow for all samples. (Right, we do support using an existing index in the workflow templates, which is in the optional configuration section. )

I will work on supporting patching in simpleaf natively when I am done with the project in my hand.

Best,
Dongze

[rob-p]

Hi @AndrewSkelton,

I'm happy to say that after some substantial design and work, @DongzeHE and I have a solution to address this. In simpleaf 0.15.0, we introduce the idea of "patching" templates. The detailed documentation surrounding patching is available here, but in short, the idea would be that you build your index once, then you fill out a workflow template pointing to this index. Finally, you prepare a "sample sheet" with all of the samples on which you want to run the workflow. Then, the patch command will generate many manifests (one per row in the sample sheet). Each of these manifests is an isolated execution to run the workflow on the given sample. You can then run these with the workflow run command, and, of course, you can orchestrate that execution however you want (looping through them one-by-one in a shell, with GNU parallel, with your cluster's job submission server, or even with a workflow system like nextflow).

Hopefully, this new feature addresses your needs. Please let @DongzeHE and I know if you have any questions, run into any trouble, or have any feedback!

Thanks!
Rob

[AndrewSkelton]

Thanks a lot @rob-p & @DongzeHE! Apologies for the lack of participation thus far (throwing down more problems than solutions!), but for some reason I didn't get the github notifications the thread had been expanded. I'll be picking up some experiments in this space in the next couple of weeks, so happy to give the solutions a test drive and feed back. Thanks for the hard work and responsiveness, it's really refreshing 👍

Andrew