Expressions vs. string interpolation (<strike>paths</strike>)

[jorgenj]

What would you like to be added:

There are really 3 concerns at hand in this issue:

1. In situations where a field expects an expression (ie; actionDataFilter.result) is it necessary for the expression text to be surrounded by ${}.
2. In situations where the expected type of result of an expression is unknown, a user should be able to clearly indicate whether they expect string interpolation or json data to be returned (for example; specifying arguments to a function invocation)
3. Some data-filter fields return data, and others return paths where other data should be merged (actionDataFilter.result vs. actionDataFilter.toStateData). These different concepts should be clarified, and in particular the "path" type fields should have some examples talking about the complex ways in which it's expected to work.

Fix all the places where an 'expression' is used to specify a location in a json structure where data should be inserted/merged.

Why is this needed:

Regarding string interpolation vs. workflow expressions, if both use-cases are basically a jq expression in a string surrounded by ${}, then there are situations where it's unclear which is intended. Imho, for fields where the purpose of the field is to accept an expression, the DSL should not require the ${}, and that ${} should be reserved to indicate string interpolation is expected.

Recent changes to actionDataFilter confusingly use 'expression' for 'toStateData' attribute, and there are presumably many other places in the spec that also need to be fixed.

Expressions (denoted by a string with ${...}) typically extract or transform some values. On the other hand, a property like actionDataFilter.toStateData is expecting the user to indicate a 'path' or location in the state data where the results should be inserted/merged. It doesn't make sense to call this an 'expression', or to have the user style it like an expression. The proposal is to fix the descriptions, and remove the surrounding ${} from all of the examples.

[tsurdilo]

i dont really get this at all, but others maybe do so please chime in. for me its much less confusing just to say "if a string is supposed to be evaluated as an expression by the runtime put it inside ${ ... }".

[jorgenj]

See my comments here on 'interpolation':

#286 (comment)

[jorgenj]

Reposting this here for clarity:

Consider this example from bash:

export FOO=BAR
export ${FOO}=BAR

env

What do you expect the output of env to include?

Now consider the following example with how the spec currently handles this:

toStateData: ${ .foo }

Given the following action results:

baz: 1

Given the existing state data of:

foo: .bar

Please take a moment to consider how a user might think that they should be getting the following output because the expression ${.foo} on state data in other contexts would produce a string with the value .bar:

foo: .bar
bar: 
  baz: 1

Overall, I think there's something wrong about the way the spec is using the '${}'. In any other language I've seen, something like that is used for variable interpolation embedded in a string. In fact I think I saw examples where that's how it's being used (for example : Some string ${.value1} with ${.value2} used as a parameter for a function call).

I would propose the following be considered:

• In situations where a string value should be produced, we support embedded interpolation of expressions via ${}, where the result will be a string representation of whatever is returned by the embedded expression.
• • toStateData is one such case, the result should be a string indicating a location in the stateData where the results should be inserted. An example value should be .foo, but we could allow interpolation like .foo.${.someOtherValue} which might produce the actual string .foo.bar. If that were the case, then the action results would be inserted at .foo.bar.
• In situations where a json result should be produced, the user must specify a string containing the text of a jq expression to be evaluated, but it should not include the ${} because we're not producing a string and they're not doing variable interpolation.
• • ActionDataFilter would be an example here, it should produce a json value. The user should indicate that via some jq syntax like .foo (note that there's no ${} because we're not doing interpolation)

@tsurdilo & @neuroglia Could you please help me understand your point of view on this?

[jorgenj]

Here's an snippet from the brigadier example that I think illustrates why this is confusing the way ${} are being used:

- name: NextEventState
  type: event
  onEvents:
  - eventRefs:
    - nextEvent
    eventDataFilter:
      toStateData: "${ .nextEvent }"
    actions:
    - name: consoleLogAction
      functionRef:
        refName: consoleLogFunction
        arguments:
          log: "fired ${ .nextEvent.data.type } caused by ${ .nextEvent.data.cause.event }"

Imagine you're a user reading this spec for the first time and you see this example. Clearly the last line is trying to do some variable interpolation, but then...what's this toStateData supposed to return if these ${} indicate interpolation? A new user might guess that it's going to use the value returned from .nextEvent as the location in stateData to insert the filtered event data?

[jorgenj]

Not getting a lot of response here, so I'll try again with an example workflow definition where the expression language feature is very ambiguous on how it should work.

Given the following workflow definition

...
states:
- name: MyInjectState
  type: inject
  data: 
      a: 
         sub1: "foo"
      b:
      - "bar"
      - "baz"
      c: aString
  transition: MyOperationState
- name: MyOperationState
  type: operation
  actions:
  - functionRef
     refName: myFunction
     arguments:
       f1: ${.a}
       f2: ${.b}
       f3: interpolating ${.a}, ${.b} and ${.c}
  end: true

Can someone please explain what they think the payload to myFunction will contain?

[tsurdilo]

Can you please give some explanation for the example:

1. StateDataFilter has input/output properties which are type String and have to be expressions.
2. are f1, f2, f3 example expressions for the stateDataFilter "output" property?

The result would always be the result of the expression applied to the associated input data

[jorgenj]

Shoot, yah, this is a bad example. My intent is to show an example with the ambiguous examples side-by-side but this was the wrong way to express that. Reworking the example.

[jorgenj]

@tsurdilo I updated the example to something that's hopefully more correct syntax.

[jorgenj]

Here's what I would expect as the content of the arguments sent to the function:

{
  "f1": { "sub1": "foo" },
  "f2": [ "bar", "baz" ],
  "f3": "interpolating { \"sub1\": \"foo\" },  [ \"bar\", \"baz\" ], and \"aString\""
}

Is this what others would expect?

[cdavernas]

@jorgenj it's indeed the logical output that'd be produced with something equivalent in, say, js.

In jq, though, it would result in an error, because as discussed today, jq only supports explicit strings in such cases. Even concatenating a number fails without explicit cast. There probably are methods to do so, I guess: I'm no jq expert.

I agree however that this is confusing and might lead to unexpected/inconsistent behaviors across runtime.

IMHO, interpolation should be delegated to expression language, or be defined explicitly (aka with some kind of markup), or be removed entirely as a feature supported by the spec.

[jorgenj]

Thanks for your input Charles. I think I agree about the notion of potentially removing interpolation for now, I think there's some edge-cases that aren't clearly defined that will be confusing and like you hinted, it seems like a pretty complex feature for runtimes to implement.

[tsurdilo]

@jorgenj @neuroglia the results are what the expression returns, no magic.

I would just use jq so there are no issues:

"interpolating ${.b | tostring} and ${.a | tostring} and ${.c | tostring}"

Idk why string concatenation would be too difficult to implement :(

if result of expression is not a string and you want to concatenate it with a string, that would be same as in any programming language (would cause an error)

[jorgenj]

@tsurdilo

Great, so given that example, here's the equivalent jq as you suggested:

jq '{ f1: .a, f2: .b, f3: ("interpolating "+ (.b | tostring) +" and "+(.a | tostring) + " and " + (.c | tostring)) }'

In this case, some of the fields got the raw json value assigned, and in others interpolation was done.

How is a user supposed to understand when interpolation will occur, vs. the raw value being used by the workflow?

Put another way, for f1 the user gave the expression "${.a}" and got an object, not the tostring of that object. What if they really wanted the tostring of that and not the actual object?