Support for workflow/chain

[yaronha]

guys,

following the discussion last week F2F on Labels, workflows, etc. we think it would make sense to discuss chaining functions into a workflow using CloudEvents:

• the first function gets an event (CloudEvent or other) and process it
• it forwards the event to the next function (or message queue) while preserving some of the original CE data and adding additional data/metadata (e.g. workflow ID, some labels, new/original source, ..)
• there maybe different next hop end-points/queues for normal/error cases
• another function reads that event, process it, and optionally passes it to another function/queue

To enable such a use case we may need to discuss some questions like who is the source (does the middle function become a source or preserves the original), which workflow metadata we may want to define explicitly or through labels/extensions, is there a way to handle affinity/distribution (e.g. like IPv6 flow labels), etc.

wonder if this is something interesting for others as well (we wanted to piggy back on CE for our Workflow support)? and do we think it can/should be part of CE?

Yaron

[duglin]

@yaronha would this be something for the CE WG or a new project under the Serverless WG?

[yaronha]

@duglin IMO part of CE (it's definitions, not implementation)

[duglin]

definitions yes, but I would think a new work-stream spawned from the serverless WG would be a better home for it. It could end up being the same people, but I view a workflow spec as independent of CE (even though it might be a user of it).

One thing we should discuss is the timeline. Others have also mention the idea of workflow, or orchestration, so I suspect others have an interest too. But people have also mentioned function signatures, and APIs for accessing the CE (I think you mentioned this). If its the same people in all of them I have my doubts we can do them in parallel. I'll add this to the agenda for tomorrow.

[leecalcote]

OpenTracing provides relevant consideration in terms of event (span) correlation.

[markfisher]

in terms of the event metadata aspect of this, Message History as described in Enterprise Integration Patterns seems relevant: http://www.enterpriseintegrationpatterns.com/patterns/messaging/MessageHistory.html

That suggests an entry is included in the list for each "component that processes the message (including the originator)", where "originator" would be the "source" in CloudEvents terminology.

[cathyhongzhang]

The "workflow" keyword in the title could cause confusion with the "workflow" concept defined in the Serverless Whitepaper. The scope and functionality described in this issue is not the same as the "workflow" defined in the Whitepaper. Suggest to change the title to "event chain".

[markfisher]

Related to my previous comment, I think "event history" is a better description than "event chain". This is not really describing a chain of events, but rather a chain of functions through which a single logical event has passed. So, from that event's perspective, it is a history of multiple hops, correlating with a trace comprised of multiple spans.

[duglin]

@yaronha do we still need this issue open given we've agreed to have "Workflows / Function Composition" as our next potential WG work item?

[yaronha]

@duglin i assume we can close, may want to ref to it so we wont lose the content

[duglin]

Added a ref from: cncf/wg-serverless#55
and closing