Automatically-generated (declarative) workflows for arbitrary interface types

[aszs]

No description provided.

[lauwers]

In order to automatically create workflows, we need to define:

1. State machines for the interfaces defined for nodes and relationships. This is described in State machine for interfaces #147
2. How are relationship operations interleaved with operations on the source and target nodes of these relationships.

The syntax for interleaving can get fairly convoluted. The following code snippet is a possible example of such syntax:

interface_types:

  Standard:
    attributes:
      state:
        type: string
        constraints:
          $valid_values:
            - $value: []
            - - initial
              - created
              - configured
              - started
              - deleted
              - error
    operations:
      create:
        preconditions:
          $equal:
            - $get_attribute: [ INTERFACE, state ]
            - initial
        on_success:
          $set_attribute: [ INTERFACE, state, created ]
        on_failure:
          $set_attribute: [ INTERFACE, state, error ]
      configure:
        preconditions:
          $equal:
            - $get_attribute: [ INTERFACE, state ]
            - created
        on_success:
          $set_attribute: [ INTERFACE, state, configured ]
        on_failure:
          $set_attribute: [ INTERFACE, state, error ]
      start:
        preconditions:
          $equal:
            - $get_attribute: [ INTERFACE, state ]
            - configured
        on_success:
          $set_attribute: [ INTERFACE, state, started ]
        on_failure:
          $set_attribute: [ INTERFACE, state, error ]
      stop:
        preconditions:
          $equal:
            - $get_attribute: [ INTERFACE, state ]
            - started
        on_success:
          $set_attribute: [ INTERFACE, state, stopped ]
        on_failure:
          $set_attribute: [ INTERFACE, state, error ]
      delete:
        preconditions:
          $equal:
            - $get_attribute: [ INTERFACE, stopped ]
            - created
        on_success:
          $set_attribute: [ INTERFACE, state, deleted ]
        on_failure:
          $set_attribute: [ INTERFACE, state, error ]

  Configure:
    attributes:
      source_state:
        type: string
        constraints:
          $valid_values:
            - $value: []
            - - initial
              - configured
              - established
              - added
              - removed
              - error
      target_state:
        type: string
        constraints:
          $valid_values:
            - $value: []
            - - initial
              - configured
              - established
              - added
              - removed
              - error
    operations:
      pre_configure_source:
        preconditions:
          $equal:
            - $get_attribute: [ INTERFACE, source_state ]
            - initial
        on_success:
          $set_attribute: [ INTERFACE, source_state, configured ]
        on_failure:
          $set_attribute: [ INTERFACE, source_state, error ]
      pre_configure_target:
        preconditions:
          $equal:
            - $get_attribute: [ INTERFACE, target_state ]
            - initial
        on_success:
          $set_attribute: [ INTERFACE, target_state, configured ]
        on_failure:
          $set_attribute: [ INTERFACE, target_state, error ]
      post_configure_source:
        preconditions:
          $equal:
            - $get_attribute: [ INTERFACE, source_state ]
            - configured
        on_success:
          $set_attribute: [ INTERFACE, source_state, established ]
        on_failure:
          $set_attribute: [ INTERFACE, source_state, error ]
      post_configure_target:
        preconditions:
          $equal:
            - $get_attribute: [ INTERFACE, target_state ]
            - configured
        on_success:
          $set_attribute: [ INTERFACE, target_state, established ]
        on_failure:
          $set_attribute: [ INTERFACE, target_state, error ]
      add_target:
        preconditions:
          $equal:
            - $get_attribute: [ INTERFACE, target_state ]
            - established
        on_success:
          $set_attribute: [ INTERFACE, target_state, added ]
        on_failure:
          $set_attribute: [ INTERFACE, target_state, error ]
      add_source:
        preconditions:
          $equal:
            - $get_attribute: [ INTERFACE, source_state ]
            - established
        on_success:
          $set_attribute: [ INTERFACE, source_state, added ]
        on_failure:
          $set_attribute: [ INTERFACE, source_state, error ]
      remove_target:
        preconditions:
          $equal:
            - $get_attribute: [ INTERFACE, target_state ]
            - added
        on_success:
          $set_attribute: [ INTERFACE, target_state, removed ]
        on_failure:
          $set_attribute: [ INTERFACE, target_state, error ]

capability_types:
  Service:
    description: Can provide service.

relationship_types:
  DependsOn:
    interfaces:
      configure:
        type: Configure

node_types:

  Server:
    interface:
      standard:
        type: Standard
        operations:
          configure:
            preconditions:
              $equal:
                - $get_attribute: [ SELF, CAPABILITY, service, SOURCE, INTERFACE, configure, target_state]
                - initial
          start:
            preconditions:
              $equal:
                - $get_attribute: [ SELF, CAPABILITY, service, SOURCE, INTERFACE, configure, target_state]
                - configured
        
  Client:
    requirements:
      - server:
          capability: Service
          node: Server
          relationship:
            type: DependsOn
            interfaces:
              configure:
                operations:
                  pre_configure_source:
                    preconditions:
                      $equal:
                        - $get_attribute: [ SELF, SOURCE, INTERFACE, standard, state ]
                        - created
                  pre_configure_target:
                    preconditions:
                      $equal:
                        - $get_attribute: [ SELF, TARGET, INTERFACE, standard, state ]
                        - created
                  post_configure_source:
                    preconditions:
                      $equal:
                        - $get_attribute: [ SELF, SOURCE, INTERFACE, standard, state ]
                        - configured
                  post_configure_target:
                    preconditions:
                      $equal:
                        - $get_attribute: [ SELF, TARGET, INTERFACE, standard, state ]
                        - configured
                  add_source:
                    preconditions:
                      $equal:
                        - $get_attribute: [ SELF, SOURCE, INTERFACE, standard, state ]
                        - started
                  add_target:
                    preconditions:
                      $equal:
                        - $get_attribute: [ SELF, TARGET, INTERFACE, standard, state ]
                        - started
    interfaces:
      standard:
        type: Standard
        operations:
          create:
            preconditions:
              $equal:
                - $get_attribute: [ SELF, RELATIONSHIP, server, TARGET, INTERFACE, standard, state]
                - started
          configure:
            preconditions:
              $equal:
                - $get_attribute: [ SELF, RELATIONSHIP, server, INTERFACE, configure, source_state]
                - initial
          start:
            preconditions:
              $equal:
                - $get_attribute: [ SELF, RELATIONSHIP, server, INTERFACE, configure, source_state]
                - configured

service_template:
  workflows:
    deploy:
      intended_state: Standard:started
      on_error:
        LifecycleManagement:
          failed: rollback
    rollback:
      intended_state: Standard:initial

[lauwers]

The example above shows code that is not very reusable. TOSCA v1.3 addresses this by advertising the Standard interface type in the Root node type and advertising the Configure interface type in the Root relationship type. The following code tries to mimic this using the proposed v2.0 interface grammar:

interface_types:
  Standard:
    attributes:
      state:
        type: string
        constraints:
          $valid_values:
            - $value: []
            - - initial
              - created
              - configured
              - started
              - deleted
              - error
    operations:
      create:
        preconditions:
          $equal:
            - $get_attribute: [ INTERFACE, state ]
            - initial
        on_success:
          $set_attribute: [ INTERFACE, state, created ]
        on_failure:
          $set_attribute: [ INTERFACE, state, error ]
      configure:
        preconditions:
          $equal:
            - $get_attribute: [ INTERFACE, state ]
            - created
        on_success:
          $set_attribute: [ INTERFACE, state, configured ]
        on_failure:
          $set_attribute: [ INTERFACE, state, error ]
      start:
        preconditions:
          $equal:
            - $get_attribute: [ INTERFACE, state ]
            - configured
        on_success:
          $set_attribute: [ INTERFACE, state, started ]
        on_failure:
          $set_attribute: [ INTERFACE, state, error ]
      stop:
        preconditions:
          $equal:
            - $get_attribute: [ INTERFACE, state ]
            - started
        on_success:
          $set_attribute: [ INTERFACE, state, stopped ]
        on_failure:
          $set_attribute: [ INTERFACE, state, error ]
      delete:
        preconditions:
          $equal:
            - $get_attribute: [ INTERFACE, stopped ]
            - created
        on_success:
          $set_attribute: [ INTERFACE, state, deleted ]
        on_failure:
          $set_attribute: [ INTERFACE, state, error ]
  Configure:
    attributes:
      source_state:
        type: string
        constraints:
          $valid_values:
            - $value: []
            - - initial
              - configured
              - established
              - added
              - removed
              - error
      target_state:
        type: string
        constraints:
          $valid_values:
            - $value: []
            - - initial
              - configured
              - established
              - added
              - removed
              - error
    operations:
      pre_configure_source:
        preconditions:
          $equal:
            - $get_attribute: [ INTERFACE, source_state ]
            - initial
        on_success:
          $set_attribute: [ INTERFACE, source_state, configured ]
        on_failure:
          $set_attribute: [ INTERFACE, source_state, error ]
      pre_configure_target:
        preconditions:
          $equal:
            - $get_attribute: [ INTERFACE, target_state ]
            - initial
        on_success:
          $set_attribute: [ INTERFACE, target_state, configured ]
        on_failure:
          $set_attribute: [ INTERFACE, target_state, error ]
      post_configure_source:
        preconditions:
          $equal:
            - $get_attribute: [ INTERFACE, source_state ]
            - configured
        on_success:
          $set_attribute: [ INTERFACE, source_state, established ]
        on_failure:
          $set_attribute: [ INTERFACE, source_state, error ]
      post_configure_target:
        preconditions:
          $equal:
            - $get_attribute: [ INTERFACE, target_state ]
            - configured
        on_success:
          $set_attribute: [ INTERFACE, target_state, established ]
        on_failure:
          $set_attribute: [ INTERFACE, target_state, error ]
      add_target:
        preconditions:
          $equal:
            - $get_attribute: [ INTERFACE, target_state ]
            - established
        on_success:
          $set_attribute: [ INTERFACE, target_state, added ]
        on_failure:
          $set_attribute: [ INTERFACE, target_state, error ]
      add_source:
        preconditions:
          $equal:
            - $get_attribute: [ INTERFACE, source_state ]
            - established
        on_success:
          $set_attribute: [ INTERFACE, source_state, added ]
        on_failure:
          $set_attribute: [ INTERFACE, source_state, error ]
      remove_target:
        preconditions:
          $equal:
            - $get_attribute: [ INTERFACE, target_state ]
            - added
        on_success:
          $set_attribute: [ INTERFACE, target_state, removed ]
        on_failure:
          $set_attribute: [ INTERFACE, target_state, error ]

relationship_types:
  Root:
    valid_source_node_types: [ Root ]
    valid_target_node_types: [ Root ]
    interfaces:
      configure:
        type: Configure
        operations:
          pre_configure_source:
            preconditions:
              $equal:
                - $get_attribute: [ SELF, SOURCE, INTERFACE, standard, state ]
                - created
          pre_configure_target:
            preconditions:
              $equal:
                - $get_attribute: [ SELF, TARGET, INTERFACE, standard, state ]
                - created
          post_configure_source:
            preconditions:
              $equal:
                - $get_attribute: [ SELF, SOURCE, INTERFACE, standard, state ]
                - configured
          post_configure_target:
            preconditions:
              $equal:
                - $get_attribute: [ SELF, TARGET, INTERFACE, standard, state ]
                - configured
          add_source:
            preconditions:
              $equal:
                - $get_attribute: [ SELF, SOURCE, INTERFACE, standard, state ]
                - started
          add_target:
            preconditions:
              $equal:
                - $get_attribute: [ SELF, TARGET, INTERFACE, standard, state ]
                - started
node_types:
  Root:
    interface:
      standard:
        type: Standard

This gets us most of the way there, since it defines how interface operations on relationships depend on the state of the Standard interface of the source and target nodes for these relationships. However, there does not seem to be a reusable way to specify how interface operations on nodes depend on the state of the Configure interface of incoming and outgoing relationships of those nodes.

[lauwers]

Perhaps we need to add a desired_state property to the interface. The state attribute allows for the definition of a state machine associated with the interface that defines the valid state transitions. The addition of a desired_state property would add the triggering logic that force the interface to transition through its state machine to get into the desired state. State transitions would be triggered whenever the desired_state property value changes or whenever the state attribute value changes.

[pmbruun]

At our last language group meeting I was asked to write a proposal or outline for "declarative workflows".

Firstly, I need to mention that HPE has a patent on this, and so a lot of the information that you ask for is in the patent text:

USPTO US-20190312794-A1 The link is not very stable - it helps if you refresh the browser after opening. Alternatively use patents.justia

While an event-based model of course can do everything, my main objection is that this is even less abstract than a classical workflow engine like those for BPEL.

"More abstract" means "more freedom for the orchestrator to decide how to do the workflow".

• At one extreme, there is "no dependency", which is what @tliron gets when the systems being orchestrated are all Cloud Native and handle their inter-dependencies autonomously. So the orchestrator can perform actions in any order whatsoever, including in parallel.
• At the other extreme, there is complete hard dependency: Do everything on node A before doing anything on node B.
• There are lots of intermediate dependency cases, but not a complete explosion in cases, though. That is what the patent is saying. You can enumerate and parameterize the meaningful cases.

It may look very generic, but by writing the event model for all the state transitions on interface states like in #153 and #147 effectively, we are making the TOSCA template designer code a workflow engine from scratch.

This is extremely complex work, something, I believe should be left to the orchestrator implementation, particularly when you factor in error handling.

The way TOSCA is structured, you can think of node/relationship-level and interface-level workflows, and obviously, the two levels need to fit together, which is not trivial if a node has multiple interfaces.

For "declarative workflows" we should focus on the lifecycle interface, connecting the lifecycles of nodes to other nodes through relationships.

A typical interface has the following "state-components":

• The state: An enumeration of possible values, like initial, created, configured, started, deleted.
  • In practice you need intermediate states as well: You could call them creating, configuring, starting, stopping, unconfiguring, destroying, or you could introduce an orthogonal transitioning boolean.
• The desired_state: Same enumeration as above, but orthogonal. This gives the lifecycle a direction - states need to progress from the current state towards the desired_state.
  • This requires a state-machine, indicating how to get from state a to state b for any pair of states.
• An error or more generally a waitfor state, capturing that some external action is required for the state machine to progress from the current to the desired state. In the HPE product we support waiting for ErrorRecovery, ManualInteraction, or ExternalResponse. In practice they work in a similar way - the orchestrator needs to stop and wait for some external event before it can progress. For now, let's focus on error, however.

Notice, that error cannot be just one of the enumeration values for state, as proposed elsewhere, because then the orchestrator would forget where it got to if the cause of the error is resolved, and it can try again. If the state is creating and error is true, then if the cause of the error (could be some system not responding, a bad value of a property, or input etc.), the system could try to progress to created by re-doing the action associated with the transition in the interface. Alternatively, the desired_state could be changed to deleted, telling the orchestrator to give up (and perhaps do something different).

Considering day 2 and teardown cases, tearing down should somehow follow the "opposite" dependencies of setup.

This is all very complicated to handle correctly, and that was just one interface, now consider the above in conjunction with multiple other nodes and relationships impacting the state-transitions and potential modifications of the desired_state and error state.

My point above was, that TOSCA template designers really shouldn't be given the task of coding that logic, including error-handling, state-transition-directions, etc. from scratch using an event-model. It is clearly doable, but just so hard to get right, that it is not helpful. Remember, our TOSCA Charter - it says, we want to make the life of the TOSCA Template Designers easy.

So my approach in the HPE product is as follows:

• Define a basic lifecycle state model instead of giving the TOSCA template designers so much freedom, that they break their necks on complexity.
• Define a set of standard rules for that state model, defining commonly occurring patterns. In the HPE product we have 5 basic types of relationships, and a number of modifier properties that you can define on the relationships, like lockstep, to further qualify the way the orchestrator creates the dependencies.

Getting cyclic dependencies is quite "easy", so the more degrees of freedom the orchestrator has for sequencing things, the better. With the above way of doing things:

• The Orchestrator can detect and report errors (see the patent for how). This is extremely useful and powerful. But with an event model, cycles cannot be detected, neither statically nor dynamically until things get stuck.
• The Orchestrator can show the dependencies graphically in a UI (see examples in the patent figures), helping users understand why it created the declarative workflow like it did.

Now all this being patented, means that TOSCA cannot really adopt it, I suppose. This is one of the reasons I have been holding back a bit. Also, of course this way is not entirely aligned with the TOSCA philosophy - that is another reason.

I have discussed with our HPE Legal team on allowing some of this to be released into OpenSource or perhaps at a nominal license fee, but that has not been successful. So for now, HPE holds and retains all rights to this and other patents, including the one on automatic creation of what TOSCA would call "dangling requirements".

[lauwers]

Yes, getting the state machine right is complicated, and we shouldn't burden template designers with this task. On the other hand, if our goal is to make TOSCA domain-independent, then we also can't really hard-code a state machine into the orchestrator. Luckily, we have a third option: we can define the state machine in domain-specific profiles. While this doesn't eliminate the complexity, it only needs to be done once for each application domain. Template designers will just pick up whatever state machine was defined in the profile they use.

[pmbruun]

I agree that it makes sense to delegate the state machine to a profile.

A principle, I have discovered over the years is: "Freedom is not always good". Sometimes, unnecessary freedom - like many different ways of doing the same thing - just adds confusion and detracts from usability.

What is problematic is that each node can have a different state machine for its lifecycle interface, even in profiles.

I have tried this in practice, and unless all lifecycle interfaces have the same state model within a single TOSCA template.

Things get extremely difficult to handle consistently when you want to automatically (declarative) generate a workflow that spans nodes with different lifecycle models.

A lifecycle state model has a setup-direction and a teardown-direction. Some state models have only two states (active/deleted), but usually we have more. Of course the state machine may decide to have different transitions for setup and teardown. It only complicates the work of assigning functionality (scripts etc.) to the state transitions, because the template designer cannot rely on teardown being the reverse of setup. For day 2 scenarios reversibility of state transitions is critical.

A solution would be to insist that:

• Lifecycle states are a linear enumeration from non-existent/deleted (0) to a most active state (n). It is my opinion and practical experience that a more complex state-machine for lifecycles does not add any value, only complexity.
• This leaves for the profile to decide the number, n, of lifecycle states it needs
• A node requiring fewer than n states can always just skip the states it doesn't need

Some templates, like those that work in a fully cloud-native environment on top of lower-level declarative orchestrators like Kubernetes may only need n to be 1. Other templates, like those working in a TMF context or with bare-metal provisioning may need additional states.

Another aspect is error-states. These are clearly orthogonal to the lifecycle states, because any state transition that invokes some script or action may fail.

An error state consists of the pair of state-numbers that corresponds to the failed state transition - the representation of that is of course up to the orchestrator. Such a representation leaves up to orchestrators, profiles and template designers to define policies for retrying or reverting the transaction. But as a fundamental model, I don't see that there are other ways of dealing with errors if you want declarative workflows as a general possibility of TOSCA and not only as a possibility for some profiles.

[lauwers]

I like the idea of enforcing linear state machines, but that might be difficult to enforce in a parser/validator, especially since an interface might have multiple state variables (e.g., one for the actual state and a second one for an error condition).