Add control flow / utility functions to the workflow engine #135

erwinvaneyk · 2018-04-17T10:43:20Z

This PR adds the following functions to the workflow engine:

Name	Description
`foreach`	Run a task for each of the elements in a list.
`http`	Invoke simple http requests.
`switch`	output a value based on the value of an expression.
`while`	Run a task until a condition evaluates to true.
`fail`	A function that always fails; which can be used to short-circuit a workflow.

Additionally, this PR...

adds (optional) server version information to the fission version command
updates the builder env and environment proxy to conform with the Fission health checking.

Before merging:

Finalize foreach + example
Add example while

soamvasani · 2018-04-17T17:08:23Z

I'm pretty excited about this one 😀 I'd love to see some examples.

soamvasani · 2018-04-19T18:15:56Z

pkg/fnenv/native/builtin/foreach.go

+)
+
+// FunctionForeach is a control flow construct to execute a certain task for each element in the provided input.
+// The tasks are executed in parallel. Foreach does not gather or store the outputs of the tasks in any way.


What is the recommendation to users for how to deal with task output?

See #135 (comment)

Not sure how that answers the question? If there is no output collected, what do we recommend to the users who want to process a set of data in an array?

soamvasani · 2018-04-19T18:25:56Z

pkg/fnenv/native/builtin/foreach.go

+//     for: [],
+//     do:
+//       run: noop
+//       inputs: "{$.???}"


Is the ??? a placeholder or the actual syntax?

Yes, not done with that one yet :)

soamvasani · 2018-04-19T18:26:21Z

pkg/fnenv/native/builtin/foreach.go

+// The tasks are executed in parallel. Foreach does not gather or store the outputs of the tasks in any way.
+//
+// For example:
+// foo:


What will $.Tasks.foo.Output be?

It is not done yet, but the idea was for foreach that there is no output (see description above this example). I am open to suggestions though :)

soamvasani · 2018-04-19T19:19:12Z

Bringing the output discussion into the comments here. I think we need to give users some way to run "map", i.e. run a task on each element of an array and collect the output into another array. That doesn't have to be a builtin, but there needs to be a way for the user to implement that.

soamvasani · 2018-04-19T19:19:40Z

cc @MadisonHope comments welcome

soamvasani · 2018-04-20T18:06:16Z

@erwinvaneyk I see lots of code commits, but can we have a discussion before merging this to make sure these new builtins actually solve the problem they're intended to solve?

erwinvaneyk · 2018-04-25T12:47:02Z

@soamvasani sure, I am not merging this until your and Madison's 👍

soamvasani · 2018-04-30T16:33:41Z

Updating from slack conversation, @erwinvaneyk's adding a flag to collect output of tasks in foreach.

ghost · 2018-05-12T21:54:01Z

I run this branch to solve some other problems on 0.2 and I couldn't make workflow send body as json to function (it does on 0.2 successfully and flask can parse request and return json response without any problem).

This is what I did extra for forcing to pass json but still not working:

....
  helloworld:
    run: task
    inputs: 
      content_type: 'application/json'
      default:
        hello: "Hey"
        world: "universe"
      headers:
        "Content-Type": "application/json"

It sends binary value after whatever I do: b'\n\x1c\n\nworld\x12\x0e\n\x06string\x12\x04universe\n\x1a\n\thello\x12\r\n\x06string\x12\x03universe'

ghost · 2018-05-13T12:20:45Z

As I understand from code, currently it is just passing value of default (body) without parsing it to json or something.

This happens if you pass key-value to the default like:

...
    default:
       input1: "{ output('Dep1') }"
       input2: "{ output('Dep2') }"

erwinvaneyk · 2018-05-14T09:25:11Z

Hey @thenamly - I have been working on this problem offline. the content type (indeed one of the last issues that needs to be resolved before 0.3.0 release). Same goes for the forcing of the content-type. Will update the branch later today.

These includde - foreach; a task that allows looping over an array of values and execute a specific task. - fail: a task that allways fails, allowing users to shortcircuit workflow execution. - http; a utility task to execute simple HTTP calls - switch; a task that executes outputs a value based on the output of an expression. Similar to switch constructs in many languages. - while;; a task that keeps executing a task until a specific condition is true.

…ucceed

Before, structs could not be parsed into the typedvalues format, which lead to those vals to be returned as null making tasks unavailable

erwinvaneyk added the wip label Apr 17, 2018

soamvasani added this to the 0.3.0 milestone Apr 17, 2018

soamvasani reviewed Apr 19, 2018

View reviewed changes

erwinvaneyk force-pushed the scatter branch 2 times, most recently from 8819dad to 851fe31 Compare April 19, 2018 18:47

erwinvaneyk force-pushed the scatter branch from 4f8a85f to f066580 Compare April 25, 2018 12:45

erwinvaneyk mentioned this pull request May 1, 2018

Fix bash variable errors at workflow builder cmd #140

Merged

erwinvaneyk force-pushed the scatter branch 2 times, most recently from e03b51a to b29a867 Compare May 15, 2018 11:29

erwinvaneyk added 10 commits May 15, 2018 13:30

Add options to wfcli to only show client version

b6849d8

Fixed build env to conform with updated Fission API

b0578a0

Fix Kubernetes-based fnref

2bf9663

Renamed input-expressions.md to expressions

a51306b

Improved debuggability of workflow builder

17df82d

Add workflow definition check to travis

89c0ff4

Fix potential issue with controller deciding on no action

4076056

Added collection options to typedvalues implementation

5d9361a

Added examples for fail, http, switch

bcf7f49

erwinvaneyk added 16 commits May 15, 2018 13:30

Fix display of wf invocation status

9b2d963

Use typedvalue formatter to partially format inputs in expr

6de6b2c

Fixed rule bug that caused completed invocation with failed task to s…

7564ac2

…ucceed

Use fnenv as a value by default

cd4c4d2

Use zero values when mapping empty typedvalues of a specific type

e344607

Add scope inheritance to dynamic tasks

4dbb11c

Improve documentation of examples

b5d5b26

Fixed expression resolver for structs

ba8e28a

Before, structs could not be parsed into the typedvalues format, which lead to those vals to be returned as null making tasks unavailable

Add administrative documentation

1886900

Fix input and output formatting in runtimes

0922a2c

Improved printing and logging

47ed463

Prioritize the parsing of inputs

f1007ba

Piped dynamic task output when invocation completes

25d4bc5

Removed old HTTP to/from conversion

35050fd

Reduced and clarified logging

b69b5cb

Added flow helper for internal://foreach

04e8c65

erwinvaneyk force-pushed the scatter branch from b29a867 to 04e8c65 Compare May 15, 2018 11:30

erwinvaneyk added 4 commits May 17, 2018 02:02

Improved error handling + reduced verbose logging

7dd466c

Fixed prioritization of inputs

03a19d3

updated examples

7aa78c3

Added e2e some test cases

d18a61f

erwinvaneyk removed the wip label May 17, 2018

Fixed e2e to be executable

e8b96d7

erwinvaneyk force-pushed the scatter branch from 12f7d65 to a137c4f Compare May 17, 2018 09:35

Add whale examples to e2e tests

6275f9c

erwinvaneyk force-pushed the scatter branch from a137c4f to 6275f9c Compare May 17, 2018 10:00

erwinvaneyk merged commit 57a1add into master May 17, 2018

erwinvaneyk deleted the scatter branch May 17, 2018 11:17

This was referenced May 17, 2018

Document functionality of the query parser #43

Closed

Change Workflows to talk to router #84

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add control flow / utility functions to the workflow engine #135

Add control flow / utility functions to the workflow engine #135

erwinvaneyk commented Apr 17, 2018 •

edited

Loading

soamvasani commented Apr 17, 2018

soamvasani Apr 19, 2018

erwinvaneyk Apr 19, 2018

soamvasani Apr 19, 2018

soamvasani Apr 19, 2018

erwinvaneyk Apr 19, 2018

soamvasani Apr 19, 2018

erwinvaneyk Apr 19, 2018

soamvasani commented Apr 19, 2018

soamvasani commented Apr 19, 2018

soamvasani commented Apr 20, 2018

erwinvaneyk commented Apr 25, 2018

soamvasani commented Apr 30, 2018 •

edited

Loading

ghost commented May 12, 2018

ghost commented May 13, 2018 •

edited by ghost

Loading

erwinvaneyk commented May 14, 2018

Add control flow / utility functions to the workflow engine #135

Add control flow / utility functions to the workflow engine #135

Conversation

erwinvaneyk commented Apr 17, 2018 • edited Loading

soamvasani commented Apr 17, 2018

soamvasani Apr 19, 2018

Choose a reason for hiding this comment

erwinvaneyk Apr 19, 2018

Choose a reason for hiding this comment

soamvasani Apr 19, 2018

Choose a reason for hiding this comment

soamvasani Apr 19, 2018

Choose a reason for hiding this comment

erwinvaneyk Apr 19, 2018

Choose a reason for hiding this comment

soamvasani Apr 19, 2018

Choose a reason for hiding this comment

erwinvaneyk Apr 19, 2018

Choose a reason for hiding this comment

soamvasani commented Apr 19, 2018

soamvasani commented Apr 19, 2018

soamvasani commented Apr 20, 2018

erwinvaneyk commented Apr 25, 2018

soamvasani commented Apr 30, 2018 • edited Loading

ghost commented May 12, 2018

ghost commented May 13, 2018 • edited by ghost Loading

erwinvaneyk commented May 14, 2018

erwinvaneyk commented Apr 17, 2018 •

edited

Loading

soamvasani commented Apr 30, 2018 •

edited

Loading

ghost commented May 13, 2018 •

edited by ghost

Loading