Improving TOSCA expressions and path traversal

[tliron]

In our current version of TOSCA path expressions:

1. We concern ourselves with a few entity types that contain "values" (either properties or attributes): nodes, capabilities (contained in nodes), and relationships (between a source node and a target capability).
2. Because values are instanced (associated with an instance) we only ever deal with one of these entities per expression.
3. The context of a function call is handled in three very different ways according to its call site:
   1. The get_property and get_attribute functions use the special word SELF to refer to the context in which the expression is evaluated. Actually, there are two different possible instances for SELF:
      1. Within a value assignment, SELF would the instance itself, a node representation or a relationship representation. (Side note: from the current documentation it's not clear to me if SELF is always either a node or a relationship, or whether it would refer to the current capability if it's called within a capability property value assignment. If it's the latter, then how do we "escape" the capability and refer to the containing node?)
      2. Within a node_filter, SELF is a candidate node. We are thus expecting the whole expression to be evaluated multiple times, once per candidate. There is an implicit loop. Note that the expression still refers only to one instance per call.
   2. To support data validation (what we used to call "constraints") we introduce the $value function. This refers not to the entity but the value assignment.

To support a rich event model in TOSCA with expressive trigger conditions and target entities, I think we need to enhance the grammar. It's also an opportunity to make it more unified and coherent.

Entity Referral Functions

Currently TOSCA path is only usable within value retrieval functions (e.g. $get_property). Let's separate the two uses.

These are the core graph traversal functions. There are structured as "from a starting point".

Without arguments (magic word) $self is the current single contextual entity. It always is associated with a specific entity type: nodes, relationships, capabilities, groups, policies, and interfaces. Arguments work like our current TOSCA path language. The name of the function refers to the starting point of graph traversal, not to the returned entity type, which otherwise depends on our graph expression. Examples:

# Just us (no arguments), can be of various entity types
$self

# From the current *node* (we're assuming it's called in a node property) to a capability
{ $self: [ CAPABILITY, my-capability ] }

# The current *capability* (we're assuming it's called in a capability property) to another capability in the node (`CONTAINER` moves "backwards")
{ $self: [ CONTAINER, my-other-capability ] }

# A specific node
{ $from_node: [ my-node-template ] }

# A specific capability in a specific node
{ $from_node: [ my-node-template, CAPABILITY, my-capability ] }

# An incoming relationship to that capability
{ $from_node: [ my-node-template, CAPABILITY, my-capability, RELATIONSHIP, 5 ] }

# An outgoing relationship
{ $from_node: [ my-node-template, RELATIONSHIP, my-requirement, 6 ] }

# A specific group
{ $from_group: [ my-group ] }

Note that this does elevate entities in TOSCA to become something like values (actually: references), as they are now being returned from functions. I suggested elsewhere that we can go one step further and make entity references into 1st-class values. It's not necessary for the rest of my proposal, but I'll just throw in an example here:

node_types:

  App:
    properties:
      friend:
        type: entity.node # notation is similar to "scalar-unit.bitrate" in which we have different entity reference types
        default: $self

We'd probably usually prefer to use a relationship in a case like this, but entity references can be used for event handlers (see #153), e.g. we might want to send an entire entity as an input to an event handler, or as a parameter in a user-defined function.

Value Retrieval Functions

The first argument is (almost) always an entity returned by one of the entity referral functions above, so we can do the same as we do now. Examples:

{ $get_property: [ $self, my-property ] }
{ $get_property: [ { $self: [ CAPABILITY, my-capability ] }, my-property ] }
{ $get_property: [ { $from_node: [ my-node-template ] }, my-property ] }
{ $get_property: [ { $from_node: [ my-node-template, CAPABILITY, my-capability ] }, my-property ] }

There's one function that doesn't accept an entity as the first argument, $value (for data validation):

# Just us (no arguments)
$value

# A sub-value
{ $value: [ my-field ] }

Entity Selection Functions

These gather zero or more entities of a specific type.

Each of these functions creates an implicit loop, where the calling context of the "condition" (a boolean sub-expression) is a candidate entity. If the expression is resolved to "true", the candidate is added to the selection. We would use $self within the condition expression
to refer to that candidate.

# Select nodes
{ $nodes: [ <condition> ] }

# Select groups
{ $groups: [ <condition> ] }

# Select relationships (from `$self` if it's a node, or from containing node if it's a capability)
$outgoing_relationships # all of them
{ $outgoing_relationships: [ <condition> ] }
$incoming_relationships # all of them
{ $incoming_relationships: [ <condition> ] }

# Full example
$nodes:
- $and:
  - { $is_a: [ $self, Machine ] } # new boolean function to check for base type
  - { $greater_than: [ { $get_property: { $self, my-capacity } }, 100 gb ] }
  - { $equals: [ { $get_property: { { $self: [ CAPABILITY, machine ] }, machine-type } } ], "vm" }

# Nested example: all machines with at least one app installed on them (machines are "in use")
$nodes:
- $and:
  - { $is_a: [ $self, Machine ] }
  - $greater_than:
    - $count: # counts entities in a selection
      - $incoming_relationships: [ { $is_a: [ $self, Installed ] } ]
    - 0

Entity selection functions would be used to select targets for sending events (see discussion #153). And also instead of the current node_filter grammar for requirements:

node_templates:

  my-app:
    type: App
    requirements:
    - server:
        targets: { $nodes: <condition as above> }
        capability: HasServer

Anywhere an entity selection is expected, a single entity referral can be used instead to refer to a selection of just one, for simple use cases.

Reimagining Groups

Entity selection functions introduce an interesting new way to think of groups. Maybe instead of being a static list of members they can use a selection function. Such "dynamic" groups would be recalculated every time they are referenced. They can also be used instead of repeated references to selectors:

groups:

  virtual:
    type: Machines
    members: { $nodes: <condition as above> }

node_templates:

  my-app:
    type: App
    requirements:
    - server:
        targets: { $group_members: virtual }
        capability: HasServer

But ... do we really need groups anymore? Their functionality can be replaced entirely with a node which has a "contains" relationship to its members. The new entity selection functions can allow us to do this especially easily:

node_templates:

  virtual: # this node functions like a group
    type: Machines
    requirements:
    - member:
        targets: { $nodes: <condition as above> }
        capability: Member
        relationship: Contains

  my-app:
    type: App
    requirements:
    - server:
        targets:
          $nodes:
          - { $has_relationship: [ { $from_node: [ virtual ] }, Contains, $self ] }
        capability: HasServer

(The $has_relationship function accepts a source node, a relationship type, and a target node.)

Obviously policy targets would work the same way:

policies:

- redundant:
    type: Redundancy
    targets: { $nodes: <condition as above> }

Handler Result Functions

These are functions like $all_handlers_succeeded and $any_handler_failed that I referred to in discussion #153. This post is already long, so I will elaborate on this topic in the future as we get through this beginning.

Textual Syntax?

As we grow and enrich our expression language, the current syntax is becoming painful. All those nested YAML { } and [ ] start to look like LISP soup (in a bad way). Maybe it's time to switch to a text-based expression language where we can use more familiar symbols.

Here's an example using . for path-based traversal (starting with $self) and : for property retrieval. It's the expression above written in a more manageable way:

{ $nodes: "(. is_a App) & (.:my-capacity > 100 gb) & (.CAPABILITY.machine:machine-type = 'vm')" }

We can possibly even support both syntax types for different reasons. For example the "textual syntax" might only be used for entity reference and selection (proposed name: "TOSCA Path Traversal Language" or "PTL"), the rest of the function calls would be the same:

{ $get_property: ".CAPABILITY:machine:machine-type" }

[pmbruun]

I like it.

More implications:

• Properties of type "Node" and list of Node. That is already in @tliron's examples of notifications.
• Inheritance gets a role here, because specializations of "Node" you could have type ABC, and that would hold any node-type inheriting from ABC. This is standard in object oriented languages.
• Ability to create relationships to all nodes in a node-type or list-of-node-type property
• Why only Node? Should Relationship also be a first-class value?

[tliron]

Yes, entity types (first-class reference values) are:

1. nodes
2. capabilities (contained in nodes)
3. relationships (incoming to or outgoing from nodes)
4. interfaces (contained in nodes or relationships)

Why all these? Because all of these are containers for values (properties and/or attributes) and events (only interfaces), which we need to refer to in TOSCA.

[pmbruun]

As soon as we introduce the concept that node N1 can have a property p1 that can contain one node N2 or a pn that can contain a list of nodes, [N3, N4], then p1 and pn are de-facto creating "relations" between N1 and those other nodes.

So once we have introduced such properties where nodes are first-class values, the obvious next step would be syntax to turn p1 and pn into real Relationship objects in the node representation graph - perhaps just by adding the desired relationship-type to the property definition?

There are two ways to go (perhaps for TOSCA 3.0?):

1. Forbid node-type properties, or
2. Fully embrace the concept of property-defined relationships

Embracing is what we do in HPE DSD, and it is an extremely powerful concept - and it is relatively easy for users to understand.

For example, the current substitution mapping mechanism is hard for users to understand and we have had to introduce syntax to handle cardinality of relationships between nodes in the surrounding graph to nodes inside the substitution. The problem is really that those collections of nodes have no good name that you can refer to and manipulate using functions, but with node-type properties, you have a property name, and expressing the relationships is quite easy.

[tliron]

I think our relationship graph is more powerful then just referencing other nodes via properties. We have a mechanism for satisfying requirements via relationships (Chris especially likes to leave them "dangling" for Day 1 live resolution). However, I do agree that there might be room for unifying the two concepts using node references as the underlying atom. I also agree it might be something for a future version of TOSCA. :)

[pmbruun]

@tliron You are absolutely right. This is not a question of one construct being more or less powerful than the other, but rather than the two ways of expressing relationships may be more or less convenient in different situations. The point is that if you anyway have a property that references other nodes, then turning that into relationships would be convenient.

Of course HPE DSD also has the requirement-capability matching capability, and we use both mechanisms depending on what makes most sense for the model designer.

[tliron]

This current exercise with an event model might take us on the path, which really started with us talking about a representation model. We're breaking down the TOSCA grammar to little building blocks that abstract automation atomics. It makes sense to me that a TOSCA relationship would just be a "special" list of target node reference properties.

And actually I would want it to be two-sided: a node needs access to its incoming node relationships.