Skip to content

[BUG] Parallel orchestration needs improvement  #59959

Closed
@gabrielgt

Description

@gabrielgt

Description
Execution of orchestration steps with parallel: True and dependencies among them using require does not work as expected.

Setup
I am trying to execute states B1 and B2 in one minion and C in another one using parallel: true, that is:

A __ B1 __ B2 __ D
  \_ C _________/

Dependencies among steps are indicated by require. The dependencies among steps could be more complicated and the results are still more unpredictable. But this example is simple enough to observe the problem.

Salt files can be generated with the following script: https://gist.github.com/gabrielgt/e6c5337e2bd69dfefc61cee2c06cc12c

Steps to Reproduce the behavior

  • Download script https://gist.github.com/gabrielgt/e6c5337e2bd69dfefc61cee2c06cc12c
  • Modify the script to use three different minion nodes: 'mn001', 'mn002' and 'mn003'
  • Run the script ./make_parallel_test_structure_2.sh
  • Run the orchestration: salt-run state.orch delme.service_orch -l debug 2>&1 | egrep 'Running state|Completed state'

Expected behavior
The expected behavior is B1 and C starting approximately at the same time. But what happens is that C starts about the same time as B2.

The duration of A, B1, B2 and D is 30 seconds, and the duration of C is 60 seconds, so it should last 120 seconds in total, but instead it lasts 150 seconds.

30 __ 30 __ 30 __________30
         \_ 60 _________/

Result:

$time sudo salt-run state.orch delme.service_orch -l debug 2>&1 | egrep 'Running state|Completed state'
[INFO    ] Running state [state.orchestrate] at time 18:32:14.612501
[INFO    ] Completed state [state.orchestrate] at time 18:32:14.620257 (duration_in_ms=7.748)
[INFO    ] Running state [orch.serviceA] at time 18:32:17.827567
[INFO    ] Completed state [orch.serviceA] at time 18:32:49.561936 (duration_in_ms=31734.369)
[INFO    ] Running state [state.orchestrate] at time 18:32:49.584701
[INFO    ] Completed state [state.orchestrate] at time 18:32:49.600972 (duration_in_ms=16.271)
[INFO    ] Running state [orch.serviceB1] at time 18:32:53.885257
[INFO    ] Completed state [orch.serviceB1] at time 18:33:25.768963 (duration_in_ms=31883.705)
[INFO    ] Running state [state.orchestrate] at time 18:33:25.789756
[INFO    ] Completed state [state.orchestrate] at time 18:33:25.804843 (duration_in_ms=15.088)
[INFO    ] Running state [state.orchestrate] at time 18:33:25.807449
[INFO    ] Completed state [state.orchestrate] at time 18:33:25.831102 (duration_in_ms=23.653)
[INFO    ] Running state [orch.serviceC] at time 18:33:30.419105
[INFO    ] Running state [orch.serviceB2] at time 18:33:31.251604
[INFO    ] Completed state [orch.serviceB2] at time 18:34:03.667342 (duration_in_ms=32415.737)
[INFO    ] Completed state [orch.serviceC] at time 18:34:32.920972 (duration_in_ms=62501.866)
[INFO    ] Running state [state.orchestrate] at time 18:34:32.943566
[INFO    ] Completed state [state.orchestrate] at time 18:34:32.956901 (duration_in_ms=13.336)
[INFO    ] Running state [orch.serviceD] at time 18:34:36.844001
[INFO    ] Completed state [orch.serviceD] at time 18:35:08.826227 (duration_in_ms=31982.226)

real    2m59.788s

Versions Report

Salt Version:
           Salt: 3003rc1

Additional context
This issue is a continuation of issue #55121, partially solved by issue #58976.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Bugbroken, incorrect, or confusing behaviorseverity-medium3rd level, incorrect or bad functionality, confusing and lacks a work around

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions