Draft design goals for CloudEvents specification

[ultrasaurus]

addresses #73

Signed-off-by: Sarah Allen <sarahallen@google.com>

[markpeek]

Thank you for putting this together. Some comments...

[markpeek]

Typo "then then"?

[duglin]

perhaps "rather than" ?

[ultrasaurus]

I was attempting to describe how late-binding of trigger + action allows for tight-binding when an application uses events. I believe that this is exactly what allows for observability. Open to suggestions on how to clarify that!

[markpeek]

This seems a bit too specific to cloud providers. Perhaps change the emphasis onto event systems rather than cloud providers?
The goal of the CloudEvents specification is to define interoperability of event systems that allow cloud providers, third-party tools and services to provide:

[markpeek]

My first read of this made me think we needed to define these terms. I then read further and saw you mention the WG needs to define this vocabulary. :-) Not sure how or whether to rectify.

[duglin]

Maybe its just me, but I suspect many people will never look at this doc - which is a shame because I think its very useful to help set the context for what the spec will talk about. Add to that, Mark's comment about needing to define the terms used in the following points, and my desire not to duplicate info, I'm wondering what people think about changing this PR so that it adds all of this info into the spec itself. This would serve a couple of purposes: 1) it will increase the chances of people reading it, 2) avoid duplication of very important concepts, 3) it'll make the spec more self-sufficient by beefing-up our intro (or where ever) section.

[ultrasaurus]

If other folks agree, I could split this PR into two parts:

1. the goals/non-goals => spec
2. WG actions => README with link to Goals section

[rachelmyers]

I think splitting this up into the spec and goals is a great idea. Let me know if you'd like me to help with that.

[markpeek]

Perhaps it is just me but I'm not sure I understand the distinction being made by this statement. Is this about event uniqueness from a namespace or source? Or having a common definition to allow the consumer to rely on a specific format?

[ultrasaurus]

Now that you mention it, both! I was just thinking about the former -- will update language to include both points

[duglin]

Can you wrap at 80 cols?

[duglin]

I think this line is important and correct. By listing something like "observability" I think it directly relates to the comments being made in #76 w.r.t. what is in/out of scope, what is "event worthy", etc... and I'm leaning towards making things more broad right now rather than less - and letting the specific implementation/use-case decide when to use CloudEvents. So, this line seems to align with that. No edit needed, just commenting.

[duglin]

I think this would also include: w/o knowledge of the specifics of the event itself - meaning the 'data'.

[ultrasaurus]

addressed feedback in commit

[ultrasaurus]

I was attempting to describe how late-binding of trigger + action allows for tight-binding when an application uses events. I believe that this is exactly what allows for observability. Open to suggestions on how to clarify that!

[ultrasaurus]

If other folks agree, I could split this PR into two parts:

1. the goals/non-goals => spec
2. WG actions => README with link to Goals section

[ultrasaurus]

Now that you mention it, both! I was just thinking about the former -- will update language to include both points

[duglin]

It feels like this paragraph (and the next one) should come after line 12 since that's where you introduce our goal of interop, so it seems like listing the things we need to do to achieve that interop right after it makes sense.

[duglin]

I would merge this paragraph with the previous one and just have one list of "goals" rather than split it into two. I would also avoid defining the terms here since we're going to do that later on (if we merge this into the spec). I would just state the goal itself - meaning, something like: Define the common vocabulary of an eventing system

[clemensv]

To me the key question for inclusion of a definition is: How does it help interoperability to attempt a shared definition of higher level constructs?

"Action" can be anything that a system might do upon receipt of an event; that includes ignoring it meaning taking no action. A "Trigger" really is a private concern of the party emitting an event; that may happen based on a deterministic schedule or based on an AI driven decision logic that no human can write down. I think our abstract notion of "occurrence" is a good one. It is on the emitting system's documentation to clarify under what circumstances events are being emitted.

In my view, neither a shared notion of "trigger" nor a shared notion of "action" are helping with the goal of semantic interoperability (i.e. semantically aligned event properties) and wire interoperability. A trigger may be the cause and an action may be the consequence of an event flow, but we can define the event flow and the context of that event flow without being specific about either.

For us to state that

a) An event is emitted by a source.
b) An event MAY be routed and dispatched by intermediaries.
c) An event can be obtained by an interested subscriber to either the source or an intermediary.

will avoid getting into specifics of why an app decides to emit an event, will avoid getting into specifics of why a receiver is interested in that event, and yet still motivate that we need enough descriptiveness in the metadata for matching up source and subscriber in some event distribution middleware.

[duglin]

I have a simpler rule... if we use a term (as a CloudEvent term and not in a traditional English usage) then we need to define it. So, if in the body of the spec we talk about 'actions' or 'triggers' as CloudEvent things then we need to define them. If we don't use those words then we don't need to define them. Then the question becomes whether we want to use those terms or not - and that's up to the PR that tries to add them. Right now our only use of the words "action" and "trigger" are more traditional English to me and not formal terms.

[austencollins]

@duglin +1 Good rule. Our company maintains a glossary. It greatly helps alignment overall.

[clemensv]

Reading a bit further, the "tight-coupling" notion you introduce here seems to surface in the idea of "Rule" that couples "Trigger" and "Action". I commented on that terminology below, but would like to make an architectural note here, because I disagree with the presented definition.

In cloud-native systems, may they be platforms or applications, there is no "during development" once you deploy V1 into production. Event-based flows provide loose coupling between the services that make up a solution while they run. It's an operational quality. Systems can be dynamically extended at runtime, and such extensions can be torn down at runtime without affecting the core system integrity. A subscriber detaching from an event source doesn't affect the event source. A subscriber may have the choice to "source" its events directly from the emitting source or through an intermediary, and it may be able switch sources to a failover secondary or the source's permanent replacement as systems roll over to a new application version or a different platform. There's no tight-coupling either way, and that is arguably the key reason for using event-driven architectures.

That reasoning is also why I don't specifically comment on "Rule" below, because it is indeed appears to be a tight-coupling construct that ties up the concrete cause of an event being emitted and the desired concrete reaction to that event, across system boundaries. That's RPC, IMO.

[duglin]

Would it address your concern if this just said:

---

CloudEvents are typically used in distributed systems to allow for services to be loosely
coupled.

---

[duglin]

@clemensv ^^

[clemensv]

Yes.

[ultrasaurus]

In the conversations over the past couple of months, I have found that there are many different kinds of events. In the industry an event can mean:

1. any async message
2. time-series data
3. what we're talking about here

There are many messaging systems (#1) that allow services to be loosely coupled which are far more general than what I believe we're discussing here.

To address concern by @clemensv, I suggest:

CloudEvents are typically used in a distributed system to allow for services to
be loosely coupled during development, then may be configured with
tight-coupling for specific applications or dynamically configured at runtime.

Will update PR

[clemensv]

If we are achieving semantic and wire-level interoperability across systems, we have already achieved a lot.

An API definition, or an API language binding, or a dispatch model for how to map event data to code would seem to define how a user application interacts with events on either the source side or the subscriber side, and that's even another layer further out into the app from "Trigger" and "Action" as laid out above.

In the messaging space, we begrudgingly need to carry the "shared API" JMS legacy around with us because of the installed base and are doing mapping work in OASIS AMQP BINDMAP, but you could get few of us excited about building another shared API standard. JMS severely constrains access to features, it imposes a particular notion of topology, it imposes a particular notion of send/receive patterns, it imposes a particular notion of filters, and so on. The innovation now all happens on different client stacks, on in some cases, like Apache Qpid Proton or AMQP for Net, those client stacks either work across products with their own API, or are the basis for product SDKs that wrap them. That all exists interoperably without the OASIS AMQP TC having imposed an API shape on those projects.

[duglin]

@clemensv are you looking for a change or is this just a commentary?

[clemensv]

I am suggesting for qualification above to be dropped and clearly state that we're not defining APIs.

[ultrasaurus]

@clemensv I'm inclined to agree, yet others have advocated for APIs as a project of the Serverless WG. In any case, I think I can re-word so as to simply clarify it is outside of our current scope

[inlined]

I would also add a non-goal that we don't need to standardize on a single identity/access control system. (E.g. we're not requiring a single spec like OpenID or inventing a pan-cloud version of IAM). This seems to follow naturally from CNCF's "No kingmakers" rule.

[ultrasaurus]

agreed. will add.

[duglin]

s/./:/

[deissnerk]

I think it is also an important design goal to define the event attributes in a way they can be added as extension to existing protocol headers like it is done for HTTP. So it's not about introducing a full-fledged event layer.

[austencollins]

+1

[duglin]

@ultrasaurus I view this PR has having two aspects to it:
1 - outlining what the spec if for, what its trying do, the problem space, etc... and I think that should go into the spec in some kind of "overview" or "intro" section. That will help set the context for readers of the spec. Putting that into a side-doc someplace else in the repo will just be ignored - and this info is too important (IMO).
2 - process related topics - things like "the wg must do ...." and I think that info should go either into the README (like in the new "WG process" section you propose) or into the "Status" section of the spec - which is already kind of our todo list. Once those items are completed we don't need to list them any place and they can be deleted.

Net: I like the concepts but I think the placement of the text should be tweaked so that its incorporated into our existing docs rather than in another one that I suspect will be hidden from most causal readers. This will also avoid the problem of potentially duplicating definition in two docs.

[ultrasaurus]

I had moved WG process notes into README in prior commit -- just moved the design goals into the spec

[duglin]

let's just say: ... when an image is uploaded to an object store.
and leave off product specific names.

[rachelmyers]

I'll change this to an image is uploaded to a specific bucket in an object store to emphasize that this is about the individual instance

[duglin]

let's leave the product names out - just remove the parens stuff - I don't think anything is lost by doing so.

[rachelmyers]

That sounds fine; I'll make this change.

[duglin]

I think its important to talk about having implementations of the spec prior to going to v1.0.
No code, no verification, no spec :-)

[yaronha]

@ultrasaurus not sure if the goal is to define the common attributes of an event (there can be many types of events w/o common structure), rather the attributes that allow us to pass the event from the source to the function reliably, observe its movement, and decipher its content (i.e. what i call the event envelop). The interop at the event level will be achieved by (future) definition of some common event types/schema.

[rachelmyers]

The roadmap calls for creating reference implementations by v0.2, I assume with the expectation that implementations will be updated as we refine the spec.

I'm resistant to including implementations here because I see this section as being the sticking points around interoperability that the spec should address, while implementations are a demonstration of the spec working.

@duglin, How strongly do you feel that the implementations should be included here?

[duglin]

If we limit this list to just what goes into the spec then I wonder if One or more common architectures that are in active use today or planned to be built by WG members makes sense.

My initial take on this list was more along the lines of "what the WG needs to do to be successful" - some will be part of the spec, some will be things outside of the spec. Sort of like... this list defines what we're going to do, while the roadmap tells people when we'll do those things. And, of course, the non-goals (below), define what we're not going to do.

[duglin]

re: Authorization model(s)... can someone elaborate on this one? access and permissions for what?

[duglin]

re @yaronha's comment, perhaps instead of:
Common attributes of an *event* that facilitate interoperability
what about:
Common metadata to annotate an *event* that facilitates interoperability ?

[yaronha]

@duglin LGTM

[rachelmyers]

@duglin, An example use case for handling Authorization is that an Event Consumer receives an Event, and will take an action of, say, unlocking the door. First it wants to confirm that this user has permission to unlock this door.

[duglin]

@rachelmyers yup - I agree with that kind of action/check being done - what's not clear to me is whether that's part of what our spec needs to define. That feels out of scope w.r.t. "defining an event" - and gets more info the processing model of the event.

[clemensv]

I agree with the general goal of event sources and event handlers to be developed and deployed independently, but it's not clear to me how that goal can be specifically tied to this specification. You can easily sabotage this goal for yourself by expressing the event payload in a proprietary fashion that causes hard coupling between event source and handler, and there's little this spec can do to help you out of that.

The core elements of the current specification do not define or mandate a "source"/"trigger"/"action" in the sense of resources that can be "developed and deployed". Those are very reasonable naming choices for concepts of a particular implementation that is using the specification, but so are "exchange"/"binding"/"queue" (RabbitMQ), "address"/"binding"/"queue" (ActiveMQ), "topic"/"selector"/"subscriber" (JMS), "topic"/"rule"/"subscriber" (Azure Service Bus), "topic"/"filter"/"subscriber" (Azure Event Grid), etc.

• as an event publisher I want to make the event available in a way that it can be distributed to an interested audience and the audience can get at a particularly interesting subset with minimal own effort (i.e. delegate filtering to generic middleware)
• as an event consumer I want to have enough fidelity in the event metadata so that I can tell that generic middleware to pick out, and deliver a subset of events to me, and for there to be enough fidelity in the metadata for local dispatch to a handler.

I prefer for the spec not to pick favorites for what code concept might produce an event, what code concept might filter/dispatch an event, and what code concept might handle the event. We can certainly talk about concrete examples like sensors or cash registers and their respective contexts on one hand and about web hooks as event handlers on the other, but I prefer for us to stay neutral regarding the architecture model and nomenclature for how those events flow from A to B in a middleware usage scenario of this spec, precisely because there's already no shortage of those models.

(more context #112 )

[ultrasaurus]

I really like the approach of identifying user stories where different personas have different concerns.

From above:

• as an event publisher...
• as an event consumer...

I'd also like to include something about

• as a developer providing observability services...

[clemensv]

Since we're inlining the goals section into the spec, it's now odd for the section to argue for specific elements of the spec itself rather than discussing the joint interoperability objective.

[rabbah]

s/faciltate/facilitate

[rabbah]

s/CloudEvent/CloudEvents

[csantanapr]

I like that the "Non-Goals" are included it helps having a clear what is the scope of the spec.

[csantanapr]

typo s/Deisgn/Design/

[csantanapr]

Why event consumers? Is up to the serveless platform to receive the event, via a trigger fire for example.
I don't know what "new event consumer" is or would look like.
I think it should remove.

Maybe you can add about guarantees that serverless provider implementing the trigger consuming the event should meet. For example the provider should not transform the message and should do it's best to deliver the event to the function as is, (ie keep date format, keep it base64 do not decode unless user asks, etc..)

[ultrasaurus]

Imagine there is an SMS service that can receive specific kinds of common event types and knows how to transform them into natural language. "Serverless" is a bit broader than FaaS.

[csantanapr]

I don't think "developer tools and libraries" should be in scope of the specification, the specification should stick to defining how an event should look like, and not what tools or frameworks a developer would use to simulate those events in her laptop, or send the events to the trigger.
"developer tools and libraries" should be move to the section of "Non-goals, out of scope"

[duglin]

I don't think the items in this list are the things the spec (or even the wg) will do, rather this are some of the things that people should be able to do with the spec, when completed.

[csantanapr]

I agree that this important, that users would like to monitor and log events, have insight how their events and flowing, for example how many events/second, or MB/sec, ingesting and triggers firing.

But I'm afraid a spec can defined this, maybe it can have a statement that to a provider to meet with the spec their platform needs to provide end user visibility into their serverless events in terms of 1. logging events on events :-) (if an event got lost, or corrupted) 2. monitoring meaning metrics (time series data) 3. visualization might be optional since the data is there and user can have a preference 3. Alerting be able with the monitoring send alerts when a threshold or criteria is met.
This is important for devops in serverless keeping an eye on your event grid.

[duglin]

s/may/can/

[duglin]

@rachelmyers @ultrasaurus a rebase is needed

[duglin]

s/may/can/

[duglin]

s/may/can/

[duglin]

All commits need to be signed to make the DCO checker happy

[ultrasaurus]

FYI -- this was discussed at today's Serverless WG meeting -- please add comments / suggestions in Google Doc

[duglin]

missing bullet markers ("-") for each

[ultrasaurus]

FYI -- this is a step forward with high level description of goals, based on discussions this week.

The group that met live, felt there still needed to be additional work done via user stories or use cases. Thinking through next steps, I think we'll need a sequence like this:

1. PR to add the list of Personas, to the README, that will be referenced in future work. Extracted from Personas being developed down the doc
2. PR to create a user-stories.md file to track/store our guiding user-stories
3. Discussion prioritizing the user stories
4. Validate that metadata attributes enable the user stories

There was some side discussion of Personas/User Stories vs Use Cases, maybe we need both... probably merits discussion in WG meeting.

[ultrasaurus]

FYI -- that was a rebase, which had a merge conflict. My editor is set to remove whitespace at the end of lines, which is minor cosmetic change.

[duglin]

I'd switch these to "needs to" because this isn't technically part of the normative section of the spec. These don't place any formal requirements on people implementing the spec.

[duglin]

LGTM for text changes and to the proposed next steps: #101 (comment)

[austencollins]

@ultrasaurus @duglin LGTM. The original Scope section is still in the spec. Now that we have the new Road Map and the Design Goals, we should take this section out (in a separate PR).

[duglin]

@ac360 yes, once this PR is merged I was going to open one to remove, or radically update, the "Status" section

[duglin]

Approved on the 3/15 call