Use functions for constraints in order to unify expression language

[tliron]

As discussed last week, here is a proposal for how it could look.

Three concepts are introduced:

1. validation keyname (instead of constraints) that is simply a boolean expression. It must evaluate to true for the value to be valid. Any expression can be used with any function with any degree of nesting.
2. A new VAL special name that refers to the current value, which can be used as the first argument in $get_property like SELF, TARGET, etc.
3. And $VAL as a shortcut for { $get_property: [ VAL ] }

Examples:

data_types:

  # "validation" is a boolean expression that must evaluate to true for the value to be valid
  # If not specified, it's as if it was "validation: true"
  # The special name "VAL" can be used with "$get_property" to refer to the value
  Count1:
    derived_from: integer
    validation: { $greater_or_equal: [ { $get_property: [ VAL ] }, 0 ] }

  # Because the above is cumbersome, let's introduce "$VAL" (escapable) as a shortcut
  Count2:
    derived_from: integer
    validation: { $greater_or_equal: [ $VAL, 0 ] }

  # Function calls can be nested with boolean logic functions, such as "$or" and "$and"
  # and really any function can be called
  Host:
    derived_from: string
    validation:
      $or:
      - { $pattern: [ $VAL, ^((25[0-5]|(2[0-4]|1\d|[1-9]|)\d)(\.(?!$)|$)){4}$ ] } # IPv4
      - $and:
        - { $equals: [ $VAL, localhost ] }
        - { $get_input: [ allow-names ] } # boolean input type

  # Powerful validations are possible across properties in a complex type
  # (impossible with the existing constraint system)
  FrequencyRange:
    properties:
      low:
        type: scalar-unit.frequency
      high:
        type: scalar-unit.frequency
    validation:
      $greater_or_equal: [ { $get_property: [ VAL, high ] }, { $get_property: [ VAL, low ] } ]

  # The following type refers to SELF, which means it can only be used in contexts in which SELF
  # exists and has the right property (see "Compute" node type below)
  MaxRAM:
    derived_from: scalar-unit.size
    validation:
      $greater_or_equal: [ $VAL, { $get_property: [ SELF, min-ram ] } ]

node_types:

  Compute:
    properties:
      min-ram:
        type: scalar-unit.size
      max-ram:
        type: MaxRAM # this will work because we have "SELF.min-ram"

[tliron]

Feedback:

1. Some people would like to keep calling it constraints (or constrain or similar).
2. VAL is confusing, sounds like "validation". Perhaps it should be VALUE?
3. Some people don't like the $VAL shortcut. Perhaps it can just be a $get_value bare string?
4. The call site within data types does not include SELF or an ability to call get_input. A call site with those would be assigned only when used in a node or relationship context. TOSCA processors may not all be happy with referring to SELF in data types.

[tliron]

Instead of reusing $get_property, we can introduce a new TOSCA-path-aware function, $get_value (and then we don't need the special name "VAL"):

  Count1:
    derived_from: integer
    validation: { $greater_or_equal: [ { $get_value: [] }, 0 ] }

  Count2:
    derived_from: integer
    validation: { $greater_or_equal: [ $VALUE, 0 ] }

  FrequencyRange:
    properties:
      low:
        type: scalar-unit.frequency
      high:
        type: scalar-unit.frequency
    validation:
      $greater_or_equal: [ { $get_value: [ high ] }, { $get_value: [ low ] } ]

[pmjordan]

If we do decide that the shortcut is desireable then rather that it being '$VALUE' we should consider
using '$THIS_VALUE'. Although this is more verbose it makes it clearer that it means get_value with an empty set, i.e. without any path.
If you want it to be really short AND discourage further syntax extensions of this kind then it could be '$_' although I don't like that much.

I'm not totally convinced the short cut is worth it. Would the following be valid and equivalent to the first example ?

Count1:
derived_from: integer
validation: { $greater_or_equal: [ { $get_value: [Count1] }, 0 ] }

[lauwers]

We should consider making our function syntax more flexible, and adopt the following:

• Any string that starts with a $ is interpreted to be a function name. This makes $ a reserved character in TOSCA. String names that start with $ and that are not functions must be escaped.
• Functions that don't take any arguments don't need to use map syntax. For example

Count:
    derived_from: integer
    validation: { $greater_or_equal: [ $get_value, 0] }

is the same as

Count:
    derived_from: integer
    validation: { $greater_or_equal: [ { $get_value: [] }, 0] }

• Functions that take only one argument don't need to use the 'list' syntax for arguments. For example

NonZero:
    derived_from: integer
    validation: { $not: { $equal: [ {$get_value: [] }, 0 ] }

is the same as

NonZero:
    derived_from: integer
    validation: { $not: [ { $equal: [ {$get_value: [] }, 0 ] ] }

I think this would simplify function syntax without introducing additional parsing challenges.

[pmjordan]

The propsal from @lauwers looks good but I have one question: If I were writing a custom function which took only one argument would I have to write it so that had the capability to accept
a) the argument
and also as an alternative
b) a list with one entry which contained the argument?
if so then not so great, it would better if the TOSCA parser did that for me and passed only one of those possibilities, in which case we would have to specifiy which one.

[calincurescu]

Some input regarding what functions would be useful for both conditions and constraints and have them as supported by default (all of them boolean):

1. Functions that could work for both lists (seq) and strings:
   has_suffix(string, string)
   has_suffix(list, list)
   has_prefix(string, string)
   has_prefix(list, list)
   contains(string, string)
   contains(list, list)

2. Functions to check if is a value or key in a list/map
   has_entry (list, value) #the value is a value in the list
   has_entry (map, value) #the value is a value in the map
   has_key (map, key) #the key is a key in the map

3. Functions
   has_all_entries (list, list)
   has_all_entries (map, list)
   has_all_keys (map, list)
   has_any_entry (list, list)
   has_any_entry (map, list)
   has_any_key (map, list)

4. Boolean logic functions:
   and(a,b,c,...) # a * b * c * ...
   or(a,b,c,...) # a | b | c | ...
   not(a) # ^a
   xor(a,b) # (a * ^b) | (^a * b)

5. for later: forall and exists:
   Functions that apply another function (given as the 1st argument) to each of the elements of the list (2nd argument) and a value (3rd argument).
   e.g.:

$forall: [equal, [el1, el2, ... eln], value]]
$forany: [equal, [el1, el2, ... eln], value]]

$forall: [has_value, [el1, el2, ... eln], value]]
$exists: [has_value, [el1, el2, ... eln], value]]

Note1: we do not write equal with $ since we do not want to evaluate equal before evaluating forall
Note2: arg1 must be a string that resolves to a boolean function name, arg2 must be a list.

Non-boolean:
3. Set semantics:
union (list, ...)
intersect (list, ...)

arithmetic functions:
sum (val, ...)
subtract (val, val)
product (val, ...)
divide (val, val)

[tliron]

In today's ad hoc with proposed a general syntax for magic words in TOSCA, see issue #133.

With that in mind we proposed that the function to use to retrieve the current value in constraint contexts be $value. Examples based on the original post above:

data_types:

  # Full syntax with no arguments
  Count1:
    derived_from: integer
    validation: { $greater_or_equal: [ { $value: [] }, 0 ] }

  # Magic word
  Count2:
    derived_from: integer
    validation: { $greater_or_equal: [ $value, 0 ] }

  # Full syntax with arguments
  FrequencyRange:
    properties:
      low:
        type: scalar-unit.frequency
      high:
        type: scalar-unit.frequency
    validation:
      $greater_or_equal: [ { $value: [ high ] }, { $value: [ low ] } ]

[pmbruun]

As I understand it a custom function could in principle also be called value. At this point in time, we are at liberty to define reserved TOSCA magic words like value, but if after TOSCA 2.0 has been out for some time, and in TOSCA 2.1 we want to introduce a new magic word hello, then we risk that there are already solutions out-there that defined a custom function named hello.

So in my experience, it is a good idea to reserve a namespace for TOSCA reserved words before anyone is using the custom functions feature.

For example we could say that all reserved magic words must end in two underscores $value__ or $hello__. That way we can later safely introduce new magic words without risking backwards compatibility issues.

[lauwers]

I believe the current spec makes it impossible to define a custom function with the same name as a built-in function, since such a function would be interpreted as a refinement/augmentation of the built-in function, not an entirely new function in its own right. My understanding is that refinements can only add implementations to existing function signatures, or append new function signatures.

[tliron]

This is an important issue for us to consider -- what if you import two profiles in which each defines a function with the same name? @pmbruun has a point that perhaps functions should be namespaced, perhaps with the same ":" notation we use for types. e.g. something like $aws:server_name

[lauwers]

Yes, I thought we had previously decided that function names are namespaced. Can you think of reasons why they shouldn't?

[lauwers]

This proposal has been accepted and is documented in Section 5.4.6 of https://docs.oasis-open.org/tosca/TOSCA/v2.0/csd05/TOSCA-v2.0-csd05.html. That section shows the use of the $value function when defining constraints/validations.