Improve DAG rendering

[pditommaso]

Scenario

Nextflow supports the rendering of the workflow DAG either in html and Graphviz DOT, which in turns allows the generation in several other formats including pdf, svg, png, etc.

However the current implementation provides a very basic pictorial representational and does not clearly represents some important elements such as inputs and outputs. See an example below:

Moreover it does not allows the DAG rending without fully execute the workflow application.

Deliverable

The goal of this enhancement is to improve the Nextflow DAG rendering so that:

1. inputs and outputs are grouped together and more easily identifiable
2. use a color code to better highlight different workflow components
3. render sub-workflows and custom function/operators implemented in the new dsl
4. allow interactive navigation of the graph when the DAG is rendered in HTML format.
5. report the number of executed tasks for each node (see Write the number of parallel processes in DAG visualization report #1025)
6. add the ability to render the DAG without the need to run the workflow application (dry run mode)

The resulting graph should be compatible at pictorial level with the DAG generated by the CWL viewer, for example:

[heuermh]

Small typo: Moreover it does now → Moreover it does not

I'm interested in producing an interactive version of this graph in Cytoscape, but I understand thick clients aren't really in vogue any more. Since this issue is linked to from the GSoC page, let me know if this comment is out of scope and I'll delete it.

[pditommaso]

The html dag rendering already uses Cytoscape. What's your plan improve it? I would suggest a separate issue tho.

Small typo

Fixed. Thanks!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[pditommaso]

Interesting library for graph rendering https://github.com/kieler/elkjs/

[sfehrmann]

Might PlantUML support what you're looking for? Especially the deployment section is quite powerful, though I got the job done with the object representation (for internal use, that is)

[lukbut]

This would be beneficial for us as our engineers value this graphical representation of workflow definitions

[bentsherman]

Made some headway on this effort today. I took the workflow diagram for nf-core/fetchngs (because it is relatively small) and modified it by hand as follows:

• remove operator nodes and channel annotations, show only empty nodes for branching
• add subgraph for each workflow
• add subgraphs for workflow inputs and outputs

Already it's a huge improvement, and I think I can automate it without modifying the DAG. Next step would be to better annotate the workflow inputs and outputs, maybe keep some of the process input/output channel labels as well. I would like to be able to click on subworkflows to collapse/expand them. Mermaid has some support for interactivity, but haven't used it before.

Once I automate this part, then I'll try it on rnaseq and sarek 😄

Before:

flowchart TD
    p0[Channel.from]
    p1([splitCsv])
    p2([map])
    p3([unique])
    p4[Channel.empty]
    p5[ ]
    p6([NFCORE_FETCHNGS:SRA:SRA_IDS_TO_RUNINFO])
    p7([first])
    p8([mix])
    p9([NFCORE_FETCHNGS:SRA:SRA_RUNINFO_TO_FTP])
    p10([first])
    p11([mix])
    p12([splitCsv])
    p13([map])
    p14([unique])
    p15([first])
    p16([mix])
    p17([map])
    p18([branch])
    p19([NFCORE_FETCHNGS:SRA:SRA_FASTQ_FTP])
    p20(( ))
    p21([first])
    p22([mix])
    p23([map])
    p24[Channel.empty]
    p25([NFCORE_FETCHNGS:SRA:FASTQ_DOWNLOAD_PREFETCH_FASTERQDUMP_SRATOOLS:CUSTOM_SRATOOLSNCBISETTINGS])
    p26([mix])
    p27[ ]
    p28([NFCORE_FETCHNGS:SRA:FASTQ_DOWNLOAD_PREFETCH_FASTERQDUMP_SRATOOLS:SRATOOLS_PREFETCH])
    p29([first])
    p30([mix])
    p31[ ]
    p32([NFCORE_FETCHNGS:SRA:FASTQ_DOWNLOAD_PREFETCH_FASTERQDUMP_SRATOOLS:SRATOOLS_FASTERQDUMP])
    p33([first])
    p34([mix])
    p35([first])
    p36([mix])
    p37([mix])
    p38([map])
    p39[ ]
    p40[ ]
    p41[ ]
    p42([NFCORE_FETCHNGS:SRA:SRA_TO_SAMPLESHEET])
    p43([collect])
    p44([collect])
    p45([NFCORE_FETCHNGS:SRA:SRA_MERGE_SAMPLESHEET])
    p46(( ))
    p47([mix])
    p48([NFCORE_FETCHNGS:SRA:MULTIQC_MAPPINGS_CONFIG])
    p49(( ))
    p50([mix])
    p51([unique])
    p52([collectFile])
    p53([NFCORE_FETCHNGS:SRA:CUSTOM_DUMPSOFTWAREVERSIONS])
    p54(( ))
    p55(( ))
    p56(( ))
    p0 --> p1
    p1 --> p2
    p2 --> p3
    p3 -->|ids| p6
    p4 -->|ch_versions| p8
    p5 -->|fields| p6
    p6 --> p9
    p6 --> p7
    p7 --> p8
    p8 -->|ch_versions| p11
    p9 --> p12
    p9 --> p10
    p10 --> p11
    p11 -->|ch_versions| p16
    p12 --> p13
    p13 --> p14
    p14 -->|ch_sra_metadata| p17
    p9 --> p15
    p15 --> p16
    p16 -->|ch_versions| p22
    p17 --> p18
    p18 --> p23
    p18 --> p19
    p19 --> p37
    p19 --> p20
    p19 --> p21
    p21 --> p22
    p22 -->|ch_versions| p36
    p23 -->|ch_sra_ids| p28
    p24 -->|ch_versions| p26
    p25 -->|ch_ncbi_settings| p28
    p25 --> p26
    p26 -->|ch_versions| p30
    p27 -->|certificate| p28
    p28 --> p32
    p28 --> p29
    p29 --> p30
    p30 -->|ch_versions| p34
    p25 -->|ch_ncbi_settings| p32
    p31 -->|certificate| p32
    p32 -->|reads| p37
    p32 --> p33
    p33 --> p34
    p34 -->|versions| p35
    p35 --> p36
    p36 -->|ch_versions| p47
    p37 --> p38
    p38 -->|ch_sra_metadata| p42
    p39 -->|pipeline| p42
    p40 -->|strandedness| p42
    p41 -->|mapping_fields| p42
    p42 --> p43
    p42 --> p44
    p43 --> p45
    p44 --> p45
    p45 --> p46
    p45 --> p48
    p45 --> p47
    p47 -->|ch_versions| p50
    p48 --> p49
    p48 --> p50
    p50 -->|ch_versions| p51
    p51 --> p52
    p52 --> p53
    p53 --> p56
    p53 --> p55
    p53 --> p54

After:

flowchart TD
    subgraph inputs
    v0[Channel.from]
    v5[fields]
    v27[certificate]
    v31[certificate]
    v39[pipeline]
    v40[strandedness]
    v41[mapping_fields]
    end
    subgraph NFCORE_FETCHNGS
    subgraph SRA
    v6([SRA_IDS_TO_RUNINFO])
    v9([SRA_RUNINFO_TO_FTP])
    v19([SRA_FASTQ_FTP])
    subgraph FASTQ_DOWNLOAD_PREFETCH_FASTERQDUMP_SRATOOLS
    v25([CUSTOM_SRATOOLSNCBISETTINGS])
    v28([SRATOOLS_PREFETCH])
    v32([SRATOOLS_FASTERQDUMP])
    end
    v42([SRA_TO_SAMPLESHEET])
    v45([SRA_MERGE_SAMPLESHEET])
    v48([MULTIQC_MAPPINGS_CONFIG])
    v53([CUSTOM_DUMPSOFTWAREVERSIONS])
    v1(( ))
    v7(( ))
    v12(( ))
    v37(( ))
    v43(( ))
    v44(( ))
    end
    end
    subgraph outputs
    v20[ ]
    v46[ ]
    v49[ ]
    v54[ ]
    v55[ ]
    v56[ ]
    end
    v0 --> v1
    v5 --> v6
    v1 --> v6
    v6 --> v9
    v6 --> v7
    v9 --> v7
    v9 --> v12
    v12 --> v19
    v19 --> v20
    v19 --> v7
    v19 --> v37
    v25 --> v28
    v25 --> v32
    v25 --> v7
    v27 --> v28
    v12 --> v28
    v28 --> v32
    v28 --> v7
    v31 --> v32
    v32 --> v7
    v32 --> v37
    v39 --> v42
    v40 --> v42
    v41 --> v42
    v37 --> v42
    v42 --> v43
    v42 --> v44
    v43 --> v45
    v44 --> v45
    v45 --> v46
    v45 --> v48
    v45 --> v7
    v48 --> v49
    v48 --> v7
    v7 --> v53
    v53 --> v56
    v53 --> v55
    v53 --> v54