-
Notifications
You must be signed in to change notification settings - Fork 358
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nested scatter blocks #838
Comments
I want to(i) scatter scatters as requested by @skazakoff above (aka "nested scatter") and also(ii) re-array and scatter again (what I'm calling criss-cross scatter)As I show in this diagram: My attempt to ask WDL/Cromwell to do the first scatter-of-scatter above gives the following error:
It was unclear how I would go about criss-cross scattering. My current solution around this is to run two separate WDL workflows. Between the two workflows, I use a simple python script to organize the files for the second scatter. The first WDL workflow is per sample, while the second WDL workflow acts on all the outputs of the multiple runs of the first workflow (multiple samples). Example of workflows that I want to run within a single WDL workflowThese are scripts that run on the cloud that I would like to run within a single WDL script using the nest-scatter and criss-cross-scatter features. Currently, I run the first script per sample (1) for multiple samples, then run a helper script to organize all the outputs (1.5), and finally run the second WDL script across the multi-sample outputs per genomic interval (2). I would like to be able to scatter the samples, scatter per interval, then run a differently organized (criss-cross) scatter across all the samples per interval. (1) First WDL workflowUltimateScatterHaplotypeCaller_cloud_quicktest.wdl
UltimateScatterHaplotypeCaller_cloud_quicktest.json
(1.5) Helper commands and script to organize inputs for second workflowFirst, generate a text file of inputs by
|
Was this ever implemented? Or are we no longer planning on doing this feature? A user expressed interest in this feature here. |
Oops, this ticket shouldn't have been closed by that PR. Looks like a typo. |
Ah, okay! Thank you for the update. |
DSDE Methods would like this too. (Important for M2 development/evaluations) |
@kcibul I propose closing this ticket since I don't think it'll be implemented in the foreseeable future. There are actually two ways that this functionality already exists:
|
@cjllanwarne Which version of Cromwell supports this feature? I'm writing another similar WDL workflow. |
@sooheelee the |
I believe this is now complete in Cromwell 23, let me know if there is more to be done here. |
This is what I have implemented in order to get around this but it adds significant complexity to code. I agree that support for a nested scatter option is an excellent idea and would save pipeline writers a lot of time. |
By "this" do you mean nested scatter blocks or cross and subworkflows? |
@alongalor Subworkflows were completed in Cromwell 23, see the Changelog for more. |
@katevoss I think the confusion is maybe that nested scatters (which this ticket is about) are not completed. Subworkflows, which can be used as a way to obtain an equivalent result to nested scatters, are completed. |
have nested scatters seen the light yet? |
I have managed to run nested scatters on Terra in Cromwell 67. So...that's a yes! |
🥳 |
It would be great if Cromwell supported nested scatter blocks. For example:
The text was updated successfully, but these errors were encountered: