how does NF know what tasks and in what order it needs to run?

[zhangzhen]

After launching a pipeline in a terminal, the information is shown about what tasks will be run in what order, as the following figure shows:

How the NF runner know about that information? Can the information be fetched from the Weblog messages the NF runner sends?

[abhi18av]

Hi @zhangzhen ,

How the NF runner know about that information?

The NF essentially creates an execution plan in the form of a DAG (Directed Acyclic Graph) based on the dependencies of the tasks and then takes into account the infrastructure contraints to control the number of parallel task executions.

In the docs here this subtle behavior of programming in a parallel computing environment is mentioned like so -
It’s worth noting that the process convertToUpper is executed in parallel, so there’s no guarantee that the instance processing the first split (the chunk Hello) will be executed before the one processing the second split (the chunk world!).

More technically speaking it's based on the DataFlow paradigm, implemented via the GPars library.

Can the information be fetched from the Weblog messages the NF runner sends?

I'm not too sure about the weblog, however the functionality has been superceded by Tower, please refer here https://www.nextflow.io/docs/edge/config.html#scope-tower

[zhangzhen]

Thank you for your reply. Do you mean that tower has provided such information about the execution plan with the messages it sends via http post/put request? Or does NF provide sort of a way that I can take to inform the outside world the execution plan?