asynchronous workflows

[euhruska]

feature request:
Allowing entk adding more tasks before the current ones finish, allowing asynchronous workflows.

[mturilli]

Hey @euhruska, apologies for the late ping. Is this still relevant for you?

[euhruska]

Yes, still relevant. While adaptivemd supports asynchronous, it doesn't use radical.entk, but lower level implementations, which has disadvantages, just mentioning robustness and usability for domain-scientists. So implementing it on entk level would be great, but not sure how it could be actually implemented. From the user side it's probably better to require the user to define new stages, since once a stage starts running it shouldn't be modified. The user would be defining and running new stages on-the fly in the script while previous stages already run.

[mturilli]

Thanks, this is helpful.

I see two approaches:

1. Specifying an adaptive function that could determine whether to create new stages depending on partial results of an ongoing execution;
2. Steaming new stages to an ongoing workflow execution.

Option 1 is already implemented and if you think it might be interesting, you can find more details in this paper: https://arxiv.org/pdf/1801.01174.pdf
Option 2 requires a major redesign of EnTK or, more probably, a new system (building block in our lingo) designed for data/task streaming and capable of using EnTK as a workflow engine. This is in our development roadmap but it will not be available anytime soon.

[euhruska]

Maybe I could use an Ensemble of pipelines approach, use one pipeline for md stages, and several pipelines for the analysis stages. The analysis pipelines would divide up the analysis. With post_exec conditions I think I could make the analysis stages wait (run sleep stages) till a predefined md stage (in the separate md pipeline) is finished. Is that correct?
Do you think that would work? It might be easier than approach 1.

Approach 1: This might work, it would be more general than the above approach. Can you help me where in the repo the Listing 2 line requested_lambdas = AdaptiveQuadrature() refers to?

[mturilli]

Your description seems to fit our freshly developed pause/resume capability for stages and pipelines. That would enable you to pause your analysis stages instead of running sleep stages, until the MD stage has finished. This functionality is still experimental so likely buggy but available for testing if you want to try it out.

For Approach 1, I checked and the code we used for that publication is not committed yet (luckily we checked!). I have asked for it to be committed and I will let you know once it will be available.

[mturilli]

Apologies for the delay, it took a while to organize the code related to AdaptiveQuadrature(). Here you find the class definition:
https://github.com/radical-cybertools/htbac/blob/master/htbac/analysis.py#L83
and you may want to have a look also at:

• https://github.com/radical-cybertools/htbac/blob/master/htbac/namd2.ti.pl and
• https://github.com/radical-cybertools/htbac/blob/master/htbac/analysis.py#L44

[mturilli]

@andre-merzky @iparask we should discuss this to see whether we can derive useful requirements for EnTK development roadmap. After that, I think we should close this ticket.

[andre-merzky]

@mturilli : ack. I suggest we open a wiki page to track requirements like this. Like here

[mturilli]

@iparask I linked the Development page on the wiki homepage.