[Daisy] Soft dependency in the start order among the steps

[lucaskanashiro]

The way the dependencies work in Daisy is: if step B depends on step A, B will only execute after A completes successfully.
But for some cases we want A and B to start in parallel, and this is not granted.
For instance, in the shutdown image tests, we had an issue where the step (A) to shut down the instance and the step (B) to wait for the logs from A should start in parallel. But sometimes daisy starts A before B, then we end up missing some logs and B never finishes.

The expected scenario is to have both steps to start in parallel:

|--------- shutdown ------------ ... ---|
|--------- wait logs ---------|

But we have this scenario from time to time:

|----- shutdown -------|
      |--------- wait logs ---------|

which can cause issues if the log we were waiting for had already been printed.

We would like to grant that the wait step begins before the shutdown

      |--------- shutdown ---------|
|----- wait logs -------|

Our workaround, for now, adds an extra step that shut down step depends on it to start (but this is not ideal):

|- dummy -||--------- shutdown ---------|
|----------- wait logs -----------|

If we have a step dependency that can start a step after another step has started but before it is finished would help in this case. Maybe specify how much time we want to wait after the step starts or maybe we could have some modifiers in the dependency list, what do you think?

[zoran15]

Which files contain the shutdown image tests you referred to?

[lucaskanashiro]

You can see all the envolved workflows here image_test/metadata-script/shutdown-script*

[zoran15]

We can ensure step execution order follows a specific order defined in the workflow. This way, "wait logs" would always start before "shutdown".

This ordering can be implicit, based on the ordering of elements in the workflow. Or it could be explicit by introducing another element along "Dependencies".

[lucaskanashiro]

We can ensure step execution order follows a specific order defined in the workflow. This way, "wait logs" would always start before "shutdown".

Ok. And what about if we want two steps starting almost concurrently but with one of them starting a bit early? This is the case here. We do not want to make shutdown depends on wait logs because if we do that the workflow will never finish (we are waiting for the logs that will be printed when the instance is powered off). And by the documentation, all the steps that do not have dependencies will start immediately, but here seems that immediately does not mean that they will start at the same time, maybe this is why we have faced the described issue. Or they start at the same time but one step takes more time than the other to "get ready". I am not sure, that is why we brought this to discussion.

This ordering can be implicit, based on the ordering of elements in the workflow. Or it could be explicit by introducing another element along "Dependencies".

I did not know about this implicit dependency thingy. How does that work? Maybe this should be described in Workflow Configuration File Specification.

[zoran15]

Implicit ordering is probably something we should implement in the future. The idea is to have Daisy respect the order of steps in Steps and Dependencies section.

For example, if we have:
Steps {
step1:...
step2:...
}

(and no Dependencies section), then step1 should always be started before step2.

If ordering of steps is defined by Dependencies section, we would do the same. For example, if we had following dependencies:

Dependendencies: {
cleanup: [wait-for-logs, shutdown]
}

Then wait-for-logs should always start before shutdown.

This way, we don't have to extend workflow definition schema.