Add property-bucket to correlate multiple events associated with a serverless application

[cathyhongzhang]

A path string pointing to a field in the event data that can be used to correlate all the events associated with the same application instance. Different event types or different applications can have different fields in the event data/message as the correlation token. The correlation token/identifier could sit in any location of the event data.

Closes: #24

[SeanFeldman]

I'm not sure about this one.
It feels it should be an attribute with the value rather than a path in the data. Reading an attribute value for correlation ID is also more practical then reading a path and then fetch the value from the data.

Having this field on the spec that is optional and application/system specific does not appeal to me.

[duglin]

I would think that you'd want to be able to do the correlation w/o having to look at the event data itself. So I kind of assumed that if we added info like this it would be based on something in the metadata, like the event-id.

[cathyhongzhang]

Sean: are you suggesting adding an explicit attribute, e.g., correlation-token or correlation-id, to the Event Context/Metadata? If so, I am fine with it. The multi-event correlation ID information can either be fetched by reading a path into the data (in this case the path string needs to be available and read from the event metadata) or reading directly from an explicitly defined correlation-token value from the event metadata.

Doug, looks like you are thinking about the same. Correct me if I misunderstand your point.

I will update the PR soon.

[SeanFeldman]

@cathyhongzhang not quite. Correlation is a system/application concern. Not every single event will have it. As such, mandating an explicit attribute feels off. I'd think an applicaiton / system that required correlation would be a part of event medatata. And 💯% not to point back to the data to retrieve correlation value.

[inlined]

In PR #129 I proposed sources could expose labels, which allow (among other things) non-hierarchical filtering. Would that solve a superset of your needs?

[cathyhongzhang]

@SeanFeldman First we are not mandating this attribute and that is why it is defined as OPTIONAL, similar to schema-url, event-time, content-type etc. which are OPTIONAL Context Attributes. For those use cases, e.g. use case that only has a single event, the event does not need to have this correlation token/ID attribute. But there are many use cases (such as IoT use cases) that require correlation of multiple events. To support such serverless use cases, the event context must have an attribute for the event consumer to get the correlation token value to correlate multiple event instances correctly and send them to the same application/processing instance.

[SeanFeldman]

Could you articulate why would it be benefitial to pass correlation value using an optional event attribute rather than extensions @cathyhongzhang?

[cathyhongzhang]

@SeanFeldman Could you first articulate why it is good to put schema-url, event-time, content-type as optional event attributes while it is not good to put correlation as event attribute :-) ?

[lfourie]

I think that an event attribute that contains a value to correlate/associate several events from different sources is very useful, particularly in IoT use cases, where there are multiple events sources (sensors). I think this warrants inclusion as a cloudevents attribute.

[duglin]

Only writing this in this PR because it was at the top of my list, I'm not implying anything about which bucket it belongs, but for each non-REQUIRED attribute the WG will need to decide which of 3 categories it belongs:
1 - in the CloudEvents spec, marked as OPTIONAL
2 - in something like a "CloudEvents Extension Attributes" doc we produce, like proposal #126, that lists well-known extensions but have no official standing
3 - defined by a vendor or some other WG - but not this WG

The bar I believe I've heard mentioned so far is that the OPTIONAL ones in the spec should be considered used by a majority of the implementations. How do we measure that? I don't know. But a start would be for people to speak up on this PR, #63 and #76 to say whether your org will use it.

[SeanFeldman]

Discussions around schema-url, event-time, and content-type predates my participation on this WG.
Regardless, here's my thought on those.

✔️ event-time - a timestamp of when the original event has occured is logical. As a consumer of an event such as "AWS S3 object created" or "VM provisioned" I'd like to know when that took place.
❌ schema-url and content-type - the way those are defined in the current spec make little sense to me.

Still would want to hear what do you think the benefits of passing correlation value using an optional event attribute rather than extensions are. Thanks.

[cathyhongzhang]

@SeanFeldman My understanding is that Optional Event attributes refer to attributes that are needed in some use cases but not in other use cases. Use cases that involve multiple events are common and are a key category use case for serverless. One goal of this spec is to facilitate interoperability/consistency. Adding Correlation-token/id into the Event Context Attributes will help with the consistency/interoperability among different event producers and consumers. Extension is for custom field and testing, which is not the case for correlation of multiple events.

[inlined]

So I think the use case is incredibly important, though I worry that explicitly including the "correlation-id" in the event definition (as the example is described) is bordering on including destination information in the Event definition. We agreed that this was an explicit non-goal.

What if we could have free-form attributes (labels) like K8S, AWS, GCP, etc support and an event broker supported "correlation-id"s for subscription/delivery? In my mind, your IoT example event could have the label {"house": "CathyHouse"} and an Action subscribe for events with {"correlation-id": "house"}. Putting the correlation-id directly in the event seems more brittle. To use another IoT example, I might have temperature sensors. Depending on my use case, I might want to correlate the sensors on all boilers or on all devices in a warehouse.

[cathyhongzhang]

@inlined Thanks for your input! We are on the same page w.r.t most of your points. As mentioned in the commit message, different applications can have different fields in the event data/message as the correlation token/id. I think we need to solve the following issues to support this category of use cases:

1. How can we ensure that the event producer puts a label which a serverless platform or other event middle ware/consumer can use for correlation?
2. Suppose we go down the path of event labels, an event could carry multiple event labels. How could the serverless platform know which label to use for correlation? This "correlation-id" must be specified somewhere.

[inlined]

I think that different applications will want to correlate on different fields. In fact, the same application might deploy multiple actions that each listen to different event streams correlated by different fields. Thus I think a good event broker/router/whatever it's called would want to support as part of the subscription filtering and ordering constraints by any field in the event, including labels.

This does add the constraint that the IoT action must know the exact label used for "house" if it wants to correlate on houses. This does mean that some part of the system (either the Action or the subscription–a concept currently out of scope) must understand some details of the specific event provider if it's going to provide these kinds of features. I'm not entirely sure that this is unexpected or unavoidable.

[duglin]

@cathyhongzhang I added Closes: #24 to the first comment since this PR covers that issue and will then github will close that issue automatically as we resolve this issue. Hope that's ok.

[cathyhongzhang]

@duglin Sure.

[erikerikson]

Doug was kind enough to point out this PR in my submission of #161

We have found that the semantics of "FollowsFrom" (as per the Open Tracing standard) provides an appropriate framing for the asynchronous inter-relationships of facts of occurrence (i.e. events).

Correlation IDs are often paired with socket connection lifetimes and frequently relate to synchronous, transactional sequences. In the data flow that event sourcing approaches can result in, a sequence of asynchronously submitted events from multiple producers can in fact be a sequential, directly causal chain or graph. Such relationships can serve a massively vital role in helping application and system operators to answer questions of "how did happen?".

Although this could be supported by tagging via the provided extensions map, the operational importance of this information causes me to support the proposal of this as a top level attribute.

[duglin]

@cathyhongzhang what's the status of this one?

[cathyhongzhang]

@duglin Could we discuss this PR in tomorrow's meeting?

[Vlaaaaaaad]

Hi,

This was discussed at Kubecon+CloudnativeCon Face-to-Face, where I could not keep my mouth shut and promised to type out the example.

What is desired by adding a correlation id is to be able to see where an event originated from. An event source might not have all the details necessary to complete all the details about its origin. This was discussed and an example was given, but there was some confusion hence this post.

Say there is a motion sensor that is generating an event: "motion detected". Now, the motion detector knows details about itself( manufacturer, version, OS, what motion was detected and so on), but has no idea what room it is into, if it's in a house or in an apartment or what the building complex address is. A security system for a building would want to know all those details.

The motion sensor sends the event to an event gateway. The gateway knows about the motion sensor and about the environment it's in. Knowing such information it can add some more data such as the room name. The event may go to other higher level gateways which add even more information to the event. See an example below:

At the motion detector level we can create the following:

"correlation":
{
	"properties":
	{
		"name": "MotionSensorCorp Motion Plus Ultra",
		"version": "2",
		"uniqueId": "baevgr8eqdc2ww80f",
	},
	"source_data":
	{}
}

The source_data field is empty since this is the event being created the first time( there is no previous).

At the room level gateway we can further enrich the event before routing it:

"correlation":
{
	"properties":
	{
		"roomId": "21312",
		"roomName": "Living Room"
	},
	"source_data":
	{
		"properties":
		{
			"name": "MotionSensorCorp Motion Plus Ultra",
			"version": "2",
			"uniqueId": "baevgr8eqdc2ww80f",
		},
		"source_data":
		{}
	}
}

At an even higher level( apartment level) we can further enrich the event with data that night not be available at a previous level:

"correlation":
{
	"properties":
	{
		"apartmentId": "e12e312e12d",
		"apartmentNumber": "10140",
		"floor": "2",
		"address": "12, Main Street, Copenhagen, Denmark, 123456",
		"tenant": ["Jane Doe", "John Doe", "Sparky the dog"]
	},
	"source_data":
	{
		"properties":
		{
			"roomId": "21312",
			"roomName": "Living Room"
		},
		"source_data":
		{
			"properties":
			{
				"name": "MotionSensorCorp Motion Plus Ultra",
				"version": "2",
				"uniqueId": "baevgr8eqdc2ww80f",
			},
			"source_data":
			{}
		}
	}
}

And the event is now arriving at the function which executes it.

This was the whole workflow initially envisioned by the developers. But some time passes and there are now multiple buildings and the system needs to evolve. The same event can be enriched further, without any modifications to any lower levels. At the building level we would get something like:

"correlation":
{
	"properties":
	{
		"complexName": "Harmony Living 1",
		"complexBuilding": "B2",
		"address": "12, Main Street, Copenhagen, Denmark, 123456"
	},
	"source_data":
	{
		"properties":
		{
			"apartmentId": "e12e312e12d",
			"apartmentNumber": "10140",
			"floor": "2",
			"address": "12, Main Street, Copenhagen, Denmark, 123456",
			"tenant": ["Jane Doe", "John Doe", "Sparky the dog"]
		},
		"source_data":
		{
			"properties":
			{
				"roomId": "21312",
				"roomName": "Living Room"
			},
			"source_data":
			{
				"properties":
				{
					"name": "MotionSensorCorp Motion Plus Ultra",
					"version": "2",
					"uniqueId": "baevgr8eqdc2ww80f"
				},
				"source_data":
				{}
			}
		}
	}
}

This allows for easy composition and forwarding of CloudEvents that is forward-compatible. The motions sensor does not have to be concerned about anything but emitting its event to the gateway. The gateway and the applications that consume and forward this event have to care about correlation.

---

At the Kubecon+CloudnativeCon F2F it was also decided that exactly what will be in that field is out of scope for CloudEvents( maybe in scope for WG Serverless). What is in scope for CloudEvents is if this field is to be added or not.

After the discussion had at Kubecon F2F it was agreed that having this event as an extension would create some "optional but really mandatory" extensions -- this is to be avoided. What seemed to be the consensus was to have the 1 correlation field added with 2 sub-fields( say properties and source_data) added to the CloudEvents spec. This would also allow for source_data to be signed and/ or encrypted. What each event gateway on the way is doing is adding even more properties to the properties field( leaving source_data intact) and sending the event forward. Think of nesting dolls.

What this field is proposing is similar to an OpenTracing trace where each span has 1 child. Each CloudEvent would get tracing and the properties discussed would be added as context to the trace. Thomas is looking into whether OpenTracing could be used instead of adding the correlation field to CloudEvents.

It is still unclear to me if an event may be routed to multiple endpoints with different labels.

Hope I understood this correctly and that the example is clear enough. If I did understood something wrong or misused terms, please let me know.

[inlined]

While I accept that any signature system would need to sign ever-increasing blocks of scope, I dislike the nesting-dolls model of adding annotations. This exposes the network infrastructure through which the event was delivered, which can add to fragility. A good test is that adding a well-behaved gateway in the middle should not break any applications. The nesting-doll approach violates this.

Also as a small nit, though we discussed two types of property bags, we didn't discuss (and I don't agree with) putting them under a "correlation" property. Source labels/properties fulfill the requirement that events can be correlated, but aren't limited to that purpose.

I would suspect something similar to:

{
  "eventID": "123",
  "source": "homsecurity://uuid",
  "eventType": "homesecurity.motion",
  "labels": {
    "softwareversion": "1.2.3.",
    "displayName": "Joe Livingroom",
    "sensitivity": "HIGH",
  },
  "annotations": {
    "complex": "Paradise Suites",
    "unit": "1337",
  },
  "contentType": "application/json",
  "data": {
    "bestGuessReason": [
      {
        "label": "cat",
        "confidence": 0.9,
      }
    ]
  }
}

(Btw, sample events suddenly make this creepy. I'm curious whether the spec should allow for redaction at a gateway too)

[Vlaaaaaaad]

Yup, leaking network infrastructure into the event is bad. Both adding properties( be it labels or annotations) and signing the content without a nesting-doll approach is unclear to me. A list of signed stuff seems... dirty. There may be a more elegant approach, but I am missing it.

[erikerikson]

Reading this and recalling previous discussion, it appears as though there is a fusion of source and the notion of correlation discussed here.

I would expect correlation to reflect specific actions (state reports, requests for state, command issuings) taken by a source, not the source itself. #24 seems to reflect that sensibility. Expanding on the established semantics of correlation seems unwise.

[lfourie]

The use of a base structure with a bag of immutable properties created by the event source, and an annotated structure with a bag of annotations added by event GWs and brokers was also discussed at the F2F. This is similar to what is described in http://docs.oasis-open.org/amqp/core/v1.0/amqp-core-messaging-v1.0.html. A similar structure could be used for correlation labels.

[inlined]

@lfourie Though the AQMP link was interesting, was this the section that you intended to send? In a quick scan I didn't grasp how the encryption spec relates to attributes.

[johnmccabe]

strcuture -> structure

[Vlaaaaaaad]

That might be a thing. In the motion sensors example that sends an event to an apartment-level gateway and then a building-level gateway there would indeed be properties that apply to multiple events( say address). See #128 (comment) for some context and examples.

[cathyhongzhang]

Will fix the typo.
Vlad, Not sure what you mean by multiple events. I would view your example as the "same" event from the eventual event consumer's point of view (e.g. the serverless platform) , with intermediate GWs adding more info or modifying the original event info. As to whether the source is the original sender or the last mile GW, that is a separate discussion.

[johnmccabe]

@cathyhongzhang I deleted a follow-up comment asking if was worth clarifying whether properties were source specific or common to all sources. Deleted it as I'd raised it during the call without digesting @Vlaaaaaaad's detailed summary. I think that's what was being responded to, hadn't refreshed before deleting.

[markfisher]

as mentioned on WG call, this could be simply named "properties" (more consistent with parallel "extensions")

[cathyhongzhang]

will do

[deissnerk]

Are properties only added when the event is created, or can they be changed/added on the event's way through the infrastructure?

[Vlaaaaaaad]

Still being debated, see #128 (comment)

[cathyhongzhang]

I think they can be changed/added

[inlined]

I think it's worth considering calling this Correlation-Context and using the existing WC3 proposal here

I'd want to check whether middleware might strip this and whether the byte limit is too restrictive, but it has the advantage of existing semantics and software support. Debugging a complex eventing system can become much easier when visual diagnostic tools already exist to trace through events for apartment=10140.

[glennblock]

One suggestion I brought up on the last call was around schema for the properties. If the property bag is intended to really be open-ended, it seems like some level of description would be valid. I proposed JSON-LD as one possibility because it has been around a while, is in pretty wide use, and because it is very flexible / can attach to existing JSON payloads easily.

[glennblock]

Aside from that I think the overall idea of properties makes sense. Based on my own experience in similar domains, my preference is it, properties hangs off the top level. That makes it really easy for gateways / middleware to pick it up. If you bury it in extensions that makes the logic more complicated for all consumers of extensions right? Also semantically it seems wrong (to me) to put it in extensions, as it is just additional metadata, rather than an enhancement to CloudEvents itself, which is what at least I read when I hear "extensions"

[cathyhongzhang]

I don't think it is a good idea to put all the key-value pairs at the top level. That will result in too many user-defined top level attributes. It is better to put these key-value pairs under the "properties" attribute.

[glennblock]

@cathyhongzhang just to clarify when I said top-level I meant off of a "properties" field hanging at the top level, not just loose key-value pairs. I agree with your point.

[cathyhongzhang]

@glennblock Oh, then we share the same thought :-) Yes, it makes more sense to put them in the "properties"

[duglin]

I'm having a really hard time seeing much of a difference between a "properties" field and the "extensions" field. Both are maps. Both allow just about anything. Both describe something about the event (either the event itself, the source, or something else - we don't really place many restrictions). I think its going to be very hard for people picking up the spec to have a very clear understanding of when to use "properties" vs "extensions" - and that confusion will hurt its adoption, and interop.

I see these type of fields being used like this:

• someone wants to add some metadata to the CloudEvent
• they define a new field "coolCompany.myInfo" and (in an ideal world) have a doc/spec defining its semantics and its syntax (type)
• event producers who want to use this new field add it to their CloudEvents
• event consumers who want to look for this new field will have prior knowledge of what "coolCompany.myInfo" is all about and will look for it, and act upon it

Notice, I didn't say (and didn't have to say) where in the CloudEvent it appears. Deciding whether "coolCompany.myInfo" appears in "properties" or "extensions" feels like an extra step that is unnecessary. Attempting to scope each bucket of properties with extra semantics isn't needed since the definition of this field should already do that.

Additionally, if we do have separate buckets for extensions, we then have to define separate rules for how to serialize each to avoid conflicts between them when they're serialized as HTTP headers. Again, not impossible, but why? It feels like extra processing/thinking w/o a clear purpose.

And... what if one day we want to add this new property to the spec? If we treat it like all other extensions (and do #225) then no one needs to change any code - we just add it to the spec as-is. (If they define it w/o a proprietary name)

[Vlaaaaaaad]

I do agree that extensions need to be defined more clearly. I see the difference between extensions and properties from a size/ effort perspective.

In my understanding proprties are simple key-value pairs, no objects or deep nesting. Simple, quick parsing.
I see extensions as more in-depth, both in terms of actual size and in terms of logic.

A properties bag could be:

"properties" : {
    "device": "MotionSensorCorp Motion Plus Ultra",
    "version": "2",
    "address": "12, Main Street, Copenhagen, Denmark, 123456",
    "apartmentNumber": "10140",
}

An extension would be something more complex. Something that extends the CloudEvent for a specific workflow or a company:

"extensions" : {
    "myBigCorpExtension" : {
        "version" : "1.3.2",
        "data": {
            "cert" : " ... base64 encoded stuff ... ",
            "internalIdForThatSpecificThing": "481770fa-70a8-11e8-adc0-fa7ae01bbebc",
            "payloadSchema": "2.9",
            "customer": "12312312312",
            "region": "eu-east-1",
            "priority" : "low"
        }
    },
    "deliveryExtension": {
        "saveAs" : "12312312-file.bin",
        "rights" : "admin",
        "location" : "/data/projectX/bla"
    }
}

It would make sense for some values from extensions to also be promoted to the properties bag, but the main idea that properties are quick and used for routing would remain. This would kind of clarify where a value should be.

[duglin]

A "size" type of argument is odd to me. In order to do something with the value of a field you need to understand the field (its semantics, syntax, etc...). If someone defines a field that's complex (or "big") and its required in order to do proper routing, I don't think its location in the CloudEvent will matter - the receiver will either want to do that complex logic or not. Its all whether they want to support those semantics (and therefore take on the burden of the complex processing), not location in the Event. As for unknown properties, I don't think size matters since people can skip (or blindly pass-on) the raw JSON w/o parsing/understanding it.

[inlined]

IMO, I think extensions are things that an events framework might leverage (e.g. tracing, checksum verification, sampling) whereas event properties are things that an event-based application might leverage (e.g. add affinity by a property or use a property to decide which clients can access events)

[duglin]

During the 6/15 f2f we agreed to:

• Cathy will modify the PR so that it includes how this is different than extensions, and how it’s different than tracing, although the language is very similar.

[duglin]

Also agreed that:

• the value should just be a "string" for now
• its a flat list of properties
• proposed name is "ApplicationProperties"
• no duplication of property names

[glennblock]

I agree with Thomas. Properties are a loose bag of data which the receiver may or may not use. Extensions I would expect are known and defined in a spec somewhere. Also they would likely have some behavior attached to them.

…

On Mon, Jun 18, 2018 at 1:48 AM Doug Davis ***@***.***> wrote: Also agreed that: - the value should just be a "string" for now - its a flat list of properties - proposed name is "ApplicationProperties" - no duplication of property names — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#128 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAInRCIOE3lskeRrm_lYs1ptziAKmVxGks5t92lvgaJpZM4S8GsE> .

[duglin]

Actually, extensions and "properties" are similar in that both may be known, or unknown, to the receiver, both could have behavior associated with them (even if just for routing), and both will require out of band knowledge between the producer and receiver to understand them since neither are defined in the spec. Extensions will be in a non-normative doc, not part of the spec.

IMO, as of the f2f, the only diff between properties and extensions are that properties can only be strings.

[cathyhongzhang]

@duglin , Maybe I misunderstand it. I thought the consensus is that the spec will not have the "extension" in the "Context Attributes", and every new attribute will be added as a top-level one. If so, why do we need to include description on how this is different from "extension"?

[cathyhongzhang]

@duglin I am going to add clarification on the definition of property itself.

[duglin]

@cathyhongzhang while we won't have a property called "extensions" we will have the notion of extensions and each serialization/binding spec will say where those extensions go - in many cases they will be treated like all of the defined properties of the spec (I hope). But, at the f2f we asked for some text to explain why these extensions are different from the ApplicationProperties - which to some will be viewed as another extension - and that's going to be confusing w/o some clear rationale for why/how they're different.

[cathyhongzhang]

@duglin I understand your point. But going down this "explain difference" path, does it mean that every new attribute to be added should explain the difference from "extensions"? It seems the root cause is that the "extensions" attribute was NOT defined clearly in the first place and thus the confusion. Given that we will remove the "extension" (right decision ! ), there seems no comparison needed. How about we forget about "extensions" and let me try to clarify the definition of the "ApplicationProperties"? And let's see if the group has any input on better clarification or is OK with this PR/definition in this week's meeting?

[duglin]

@cathyhongzhang can you fix the DCO issue and rebase?

[duglin]

s/MANDATORY/REQUIRED/

[duglin]

I think it would be easier to just define this as a Map and say the "value" MUST be a "string" instead of an "Any". That would remove the need for most of the rules specified below because they would be implicit/obvious. But if you do keep any of the rules then they should use MUST instead of "should".

[duglin]

missing , at the end of each line