Topics/Subject Prior Art and Example

[clemensv]

We (Microsoft) believe that the technical core artifact that the WG has developed to date, the property definitions, are quite close to where the WG needs to be for an initial "stable" draft that teams can start to do some bi- or multilateral interop work on. In other organizations, there's a differentiation between a
working draft that changes day-to-day and a committee draft that has been promoted to a status where implementation makes sense. That doesn't mean at all that a subsequent draft wouldn't have breaking changes; it simply means that a quorum of WG members are willing to try out the status quo.

Allow me to substantiate this in the following, with several examples and pointers to prior art, and also some principles to apply to goals.

The goal of this WG could be a holistic definition and shared base infrastructure implementation for creation, routing, dispatch, and handling of "events". We do not think such a goal would be realistic.

At Microsoft alone, we have several dozen of such infrastructures, each with its own native notion of events, and most existing with good reason since they aim either at a particular audience or address certain architectural needs. This doesn't look different in other larger firms.

What this WG can achieve, and in large part already has achieved, is to establish consensus on a sufficient set of well-defined criteria that allow generic infrastructure to route and dispatch events to event handlers, and to enable operational monitoring of such infrastructure to track and display the events in a uniform fashion.

The primary purpose of achieving such a consensus is interoperability. The specification may also be useful for consistently wiring up the various parts of a single vendor's stack internally, but we're ultimately not pooling resources for everyone to build their own stuff better in isolation.

Interoperability has many facets, so let me call out the most fundamental interoperability goal that I believe most of the group is chasing:

With apologies to everyone whose products I'm not mentioning: An event raised in the Oracle Cloud should be routable through Azure Event Grid, with the subscriber being an IBM Cloud Function. An event raised in the Google Cloud should be routable through AWS SNS be subscriber being someone's custom web-hook handler. And with all that, the code in that web-hook handler shouldn't be looking like Yaron's example (slide 2)

Once we jointly declare victory on that goal, we can proceed to see whether there is more common ground for achieving consensus on specific categories of events, such as time-series telemetry.

Which criteria do we really need?

Which routing and dispatch criteria do we need? Whether it's headers, or properties, or fields, or metadata, an event will always carry information that answers some basic questions about the event.

The core data of an event that tells us that a light bulb was just turned off is some indicator that says "OFF", or a boolean "false", or a zero-valued bit/byte/word/int/etc: "0".

The further that value travels from the state register inside of the switch or socket, the more we need to enrich it with information that answers questions the consumer of the event will likely have, and that middleware will have on behalf of its attached subscribers/consumers as it routes the event towards
them:

• Where/in what context is the event occurring?
• What kind of event is this?
• Who emitted the event? (most often: which thing)
• When was this event emitted?
• Do you have more details?

Having answers to these basic questions matters to different audiences in different degrees.

Routing events in common middleware

A middleware router, for which there is plenty of prior art to look at, might only ever care about a coarse classification of the event, meaning answers to the first three questions, all while the middleware itself will care only minimally about semantics.

With typical MQTT broker implementations in the IoT realm, which also includes cloud platforms such as AWS IoT, IBM Watson IoT, Azure IoT Hub, or Google Cloud IoT Core, the topic space for flowing events from the devices to consumers is typically structured in first degree by device and type ("Where?"/Context), and second by message or event type ("What?"), all composed into one topic path expression. The MQTT
infrastructure itself has no notion of what a device or event is; it just matches up publishers and subscribers by a longest shared path prefix or exact match on that path.

The MQTT topic model stems from IBM MQ topic trees, which is is a more structured concept in MQ than in the lighter-weight MQTT brokers as you create administrative objects in the broker to back those trees; but the broker itself will still not gain a notion of the semantics of the
topic, just structure. In this model, like in MQTT, you will generally use a coarse classification of events/messages as in the prior examples to inform the structure of the topic tree for the purpose of getting events to where they need to be.

Since a broker-scoped, global tree structure is both tricky to scale and also difficult to manage, most other modern message brokers have a more abstract notion of topics, whereby a topic is a named entity inside an infrastructure that is associated with a context as the application sees it fit. IBM MQ calls these
"streams", other brokers call them "topics", or "address + binding", or "exchange + binding".
In the Java Message Service (JMS) standard, they are called "topic". Many of these message brokers, examples being Apache ActiveMQ, Apache Qpid, Solace, Software AG UM, Solace, IBM MQ, and Azure Service Bus, have adopted AMQP 1.0 as their primary protocol or as interop protocol, so when we're arguing with examples from AMQP, we're arguing from an established industry consensus position. The same applies for MQTT.

Conceptually identical, even if functionally a bit different, is the topic concept in Apache Kafka, and services built on it (enMasse, IBM MessageBus), Azure Event Hubs, or Azure Event Grid.

With these broker infrastructures, the it is common practice for the context association of the "topic" itself to be fairly loose. You might use just one Kafka topic or a single Event Hub for collecting all telemetry events from a fleet of hundreds of thousands of devices.

Instead of subscribing to a shared prefix on a topic tree path as you'd do with MQTT, a subscriber chooses a topic to subscribe on, and then uses some form of filter on the metadata describing the event.

With event ingest infrastructure like Kafka or Event Hubs that is commonly used to pipe into "Big Data" analytics, you might not be more selective than picking a partition, if at all. With a message or event broker, a subscriber will typically pick out events based on more context.

With Functions-as-a-Service, you might also pick out the specific event type as you want to dispatch to a specific handler.

In many implementations, the full context of an event or message is not a key criterion for middleware to consider. It is rather left to the event handler to inspect the event, determine the full context, including the originator, and then access the appropriate objects and/or data records; in those cases, it is sufficient for such information to live in the event details rather than in the metadata considered by middleware.

Mapping an example

I have several links at the bottom of examples to study. I'm going to use one of them, an Azure Application Insights "Alert" as it shows up in Azure Monitoring to illustrate a few points. I am specifically picking an example that my team had no hand in building and that's not an Event Grid event. I'll throw the whole event record in here:

{
  "caller": "Microsoft.Insights/alertRules",
  "channels": "Admin, Operation",
  "claims": {
    "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/spn": "Microsoft.Insights/alertRules"
  },
  "correlationId": "/subscriptions/mySubscriptionID/resourceGroups/myResourceGroup/providers/microsoft.insights/alertrules/myalert/incidents/L3N1YnNjcmlwdGlvbnMvZGY2MDJjOWMtN2FhMC00MDdkLWE2ZmItZWIyMGM4YmQxMTkyL3Jlc291cmNlR3JvdXBzL0NzbUV2ZW50RE9HRk9PRC1XZXN0VVMvcHJvdmlkZXJzL21pY3Jvc29mdC5pbnNpZ2h0cy9hbGVydHJ1bGVzL215YWxlcnQwNjM2MzYyMjU4NTM1MjIxOTIw",
  "description": "'Disk read LessThan 100000 ([Count]) in the last 5 minutes' has been resolved for CloudService: myResourceGroup/Production/Event.BackgroundJobsWorker.razzle (myResourceGroup)",
  "eventDataId": "149d4baf-53dc-4cf4-9e29-17de37405cd9",
  "eventName": {
    "value": "Alert",
    "localizedValue": "Alert"
  },
  "category": {
    "value": "Alert",
    "localizedValue": "Alert"
  },
  "id": "/subscriptions/mySubscriptionID/resourceGroups/myResourceGroup/providers/Microsoft.ClassicCompute/CloudServices/myService/slots/Production/roles/Event.BackgroundJobsWorker.razzle/events/149d4baf-53dc-4cf4-9e29-17de37405cd9/ticks/636362258535221920",
  "level": "Informational",
  "resourceGroupName": "myResourceGroup",
  "resourceProviderName": {
    "value": "Microsoft.ClassicCompute",
    "localizedValue": "Microsoft.ClassicCompute"
  },
  "resourceId": "/subscriptions/mySubscriptionID/resourceGroups/myResourceGroup/providers/Microsoft.ClassicCompute/CloudServices/myService/slots/Production/roles/Event.BackgroundJobsWorker.razzle",
  "resourceType": {
    "value": "Microsoft.ClassicCompute/CloudServices/slots/roles",
    "localizedValue": "Microsoft.ClassicCompute/CloudServices/slots/roles"
  },
  "operationId": "/subscriptions/mySubscriptionID/resourceGroups/myResourceGroup/providers/microsoft.insights/alertrules/myalert/incidents/L3N1YnNjcmlwdGlvbnMvZGY2MDJjOWMtN2FhMC00MDdkLWE2ZmItZWIyMGM4YmQxMTkyL3Jlc291cmNlR3JvdXBzL0NzbUV2ZW50RE9HRk9PRC1XZXN0VVMvcHJvdmlkZXJzL21pY3Jvc29mdC5pbnNpZ2h0cy9hbGVydHJ1bGVzL215YWxlcnQwNjM2MzYyMjU4NTM1MjIxOTIw",
  "operationName": {
    "value": "Microsoft.Insights/AlertRules/Resolved/Action",
    "localizedValue": "Microsoft.Insights/AlertRules/Resolved/Action"
  },
  "properties": {
    "RuleUri": "/subscriptions/mySubscriptionID/resourceGroups/myResourceGroup/providers/microsoft.insights/alertrules/myalert",
    "RuleName": "myalert",
    "RuleDescription": "",
    "Threshold": "100000",
    "WindowSizeInMinutes": "5",
    "Aggregation": "Average",
    "Operator": "LessThan",
    "MetricName": "Disk read",
    "MetricUnit": "Count"
  },
  "status": {
    "value": "Resolved",
    "localizedValue": "Resolved"
  },
  "subStatus": {
    "value": null
  },
  "eventTimestamp": "2017-07-21T09:24:13.522192Z",
  "submissionTimestamp": "2017-07-21T09:24:15.6578651Z",
  "subscriptionId": "mySubscriptionID"
}

That's a lot of stuff.

And there's code with hard dependencies on that stuff, so it's not likely that the advent of
a CloudEvents standard is causing that to be reshaped promptly. But we can use a very convenient
and proven trick: put it into a container and then promote properties.

Wrap

First, put it into a container:

{
=>  "data" :
    {
      "caller": "Microsoft.Insights/alertRules",
      ... everything else from above ...
    }
}

Now let's declare the shape of the container:

{
=>  "contentType": "application/json",
    "schema-url": "https://github.com/Azure/azure-rest-api-specs/...",
    "data" :
    {
      "caller": "Microsoft.Insights/alertRules",
      ... everything else from above ...
    }
}

Id and Timestamp

Now let's pull up the obvious things, like the time stamp and the unique id. The source event has two time stamps; I am picking the time of the event proper:

{
=>  "eventId": "/subscriptions/mySubscriptionID/resourceGroups/myResourceGroup/providers/Microsoft.ClassicCompute/CloudServices/myService/slots/Production/roles/Event.BackgroundJobsWorker.razzle/events/149d4baf-53dc-4cf4-9e29-17de37405cd9/ticks/636362258535221920",
=>  "eventTime" : "2017-07-21T09:24:13.522192Z",
    "contentType": "application/json",
    "schema-url": "https://github.com/Azure/azure-rest-api-specs/...",
    "data" :
    {
      ...
      "id": "/subscriptions/mySubscriptionID/resourceGroups/myResourceGroup/providers/Microsoft.ClassicCompute/CloudServices/myService/slots/Production/roles/Event.BackgroundJobsWorker.razzle/events/149d4baf-53dc-4cf4-9e29-17de37405cd9/ticks/636362258535221920",
      ...
      "eventTimestamp": "2017-07-21T09:24:13.522192Z",
      ...
    }
}

Event Type

Now let's start identifying what is a sufficient criterion for me to subscribe to an infrastructure to deliver that event to me, and for me to dispatch that event to a handler once I have it. We'll start with the latter.

From the example, I am picking up the non-localized operationName as my eventType. That is already nicely qualified, but to really make sure that it doesn't clash with anyone else's idea of event types, I'm going to scope this to Azure via the namespace qualifier. That qualifier will later also help to disambiguate clashing namespace concepts for the topic space.

While the event here stems from Application Insights, it's consolidated in and surfaced via Azure Monitoring using a Monitoring schema, and therefore I use that for namespace scoping. There will be no collisions of event types amongst anything that surfaces through this path, and that is a sufficient criterion for picking the namespace.

{
=>"namespace" : "azure.com",
=>"eventType": "Microsoft.Insights/AlertRules/Resolved/Action",
  "eventId": "/subscriptions/mySubscriptionID/resourceGroups/myResourceGroup/providers/Microsoft.ClassicCompute/CloudServices/myService/slots/Production/roles/Event.BackgroundJobsWorker.razzle/events/149d4baf-53dc-4cf4-9e29-17de37405cd9/ticks/636362258535221920",
    "eventTime" : "2017-07-21T09:24:13.522192Z",
    "contentType": "application/json",
    "schema-url": "https://github.com/Azure/azure-rest-api-specs/...",
    "data" :
    {
      ...
     "operationName": {
=>      "value": "Microsoft.Insights/AlertRules/Resolved/Action",
        "localizedValue": "Microsoft.Insights/AlertRules/Resolved/Action"
      },
      ...
    }
}

If I have a handler for "resolved alerts" in my code, the eventType will be a sufficient criterion for me to pass the JSON object to that function which will interpret it in detail. If that handler is an HTTP-reachable handler and I have a handler for each of the operations (the other one is "Activated", FWIW), I can set up the required dispatch paths easily with a rule that looks at this value.

Since the handler knows exactly how to interpret the event data, there's nothing more we need for dispatching. Let's now look at how we get a message to the dispatcher.

Picking a "topic"

I'm going to assume a generic pub/sub infrastructure of the kind discussed above, instead of what the Azure Monitoring product looks like. The fundamental assumption is that the event gets passed to a topic, and that there is a filtered subscription gesture to pick out the events you are interested in, ignoring eventType as a dispatcher concern.

What is the topic? A topic space is a structure based on architectural concerns of a solution and also based on how much scope and flexibility is needed for filtered subscriptions. In some cases, If you want a broader set to select from, have a topic that is broader scoped. The broadest practical scope in a larger organization like ours is typically an ownership boundary. A topic tree (or graph) is generally an element of the solution architecture.

It's important recognize the "topic" as the conceptual rendezvous point of the producer or multiple producers and an interested consumer.

For the particular use-case we considering here, I need to point out that Azure has a universal notion of "resource" that spans all manageable artifacts across all Azure service in one graph. This will be a differently structured and named concept elsewhere, and I am only referring to "resource" in this particular sense.

With that in mind, there are four fields in the event that specifically refer to the origin of the event:

  "resourceGroupName": "myResourceGroup",
  "resourceProviderName": {
    "value": "Microsoft.ClassicCompute",
    "localizedValue": "Microsoft.ClassicCompute"
  },
  "resourceType": {
    "value": "Microsoft.ClassicCompute/CloudServices/slots/roles",
    "localizedValue": "Microsoft.ClassicCompute/CloudServices/slots/roles"
  },
  "resourceId": "/subscriptions/{guid}/resourceGroups/{myResourceGroup}/providers/Microsoft.ClassicCompute/CloudServices/{myService}/slots/{Production}/roles/{Event.BackgroundJobsWorker.razzle}",

• resourceGroupName: This is a scoping construct that ring-fences several resources, typically all resources that belong to a certain solution.
• resourceProviderName: This is the name of the resource provider that writes out the event (Application Insights is a plugin inside of that resource)
• resourceType: This is a kind of resource this event (alert) is about. Here it's specifically about the alert state of whatever runs in a particular role of a deployment slot in a compute host.
• resourceId: This is the fully qualified path to the specific resource that the event (the Alert) is actually about. I marked up the Id with curly brackets so that you can see what parts are variable. This shows the "Production" slot's "Event.BackgroundJobsWorker.razzle" role.

The resourceId is a great candidate to pick for topic scoping, because it already embeds all the other information. In an MQTT system where topics are completely dynamic and longest shared-prefix matching is supported, the resourceId maps very neatly to a topic. The producer publishes there, and a subscriber can choose whether it wants messages for that exact object or for a whole sub-branch of the resource graph, i.e. for everything matching a path prefix.

With this choice there would be complete equivalence between the "resource" expression and the "topic".

A practical problem is that such an unrestricted topic space scales fairly terribly. The solution to this scaling issue is to partition the topic space into distinct addressable entities. IBM MQ made such a progression from a singular topic tree at the server level to the notion of "streams" that can host independent tree branches and that can be individually managed and scaled differently. You can see the concept of "topic" as a manageable artifact echoed in most infrastructure due to the practical limits of doing longest prefix match on unrestricted graphs.

If I'm using an event broker with a more structured topic scheme like the many examples I called out above, I might create a topic for all events emitted from either my resourceGroup, which crosses organizational boundaries and is thus a hazardous idea not for technical reasons but for org reasons, or for either all "CloudServices" in my resourceGroup or a specific one, both of which are better ideas.

I could also make a topic scoped to a single slot or a single role, but that would add very significant complexity on the publisher side while making it harder to collect events about the overall state of the service.

I'm going to go with the service-scope option here, so let's assume we have a topic specifically for /subscriptions/{guid}/resourceGroups/{myResourceGroup}/providers/Microsoft.ClassicCompute/CloudServices/{myService} that I can subscribe to.

When messages pop out into my receiving code, I should know what topic they came from, or
in other words, but saying the same: what is the scope that the "source" publisher is publishing events for - recall this being a rendezvous point.

The scope of the publisher is the "topic":

{
  "namespace": "azure.com",
=>"topic": "/subscriptions/{guid}/resourceGroups/{myResourceGroup}/providers/Microsoft.ClassicCompute/CloudServices/{myService}",
  "eventType": "Microsoft.Insights/AlertRules/Resolved/Action",
  "eventId" : "/subscriptions/mySubscriptionID/resourceGroups/myResourceGroup/providers/Microsoft.ClassicCompute/CloudServices/myService/slots/Production/roles/Event.BackgroundJobsWorker.razzle/events/149d4baf-53dc-4cf4-9e29-17de37405cd9/ticks/636362258535221920",
    "eventTime" : "2017-07-21T09:24:13.522192Z",
    "contentType": "application/json",
    "schema-url": "https://github.com/Azure/azure-rest-api-specs/...",
    "data" :
    {
      ...
    }
}

You will now notice that we're still missing some information. The publisher publishes
at the service scope, but the middleware may still have to permit for a subscriber to
select messages, or a dispatcher to pick an object, by the concrete 'slot' or 'role'.

To do that, we have to put that info into the event metadata, but we can't put it in the
"topic" because that's the publisher's scope; that is where "subject" comes in.

The publisher context ("Where?") tells us about an event at its scope, and further qualifies that by
the subject ("What in the scope?") of the eventType ("What happened?"). Thus,
I am putting the further qualification of the resource into the subject: /slots/{Production}/roles/{Event.BackgroundJobsWorker.razzle}

{
  "namespace": "azure.com",
  "topic": "/subscriptions/{guid}/resourceGroups/{myResourceGroup}/providers/Microsoft.ClassicCompute/CloudServices/{myService}",
=>"subject" : "/slots/{Production}/roles/{Event.BackgroundJobsWorker.razzle}",
  "eventType": "Microsoft.Insights/AlertRules/Resolved/Action",
  "eventId" : "/subscriptions/mySubscriptionID/resourceGroups/myResourceGroup/providers/Microsoft.ClassicCompute/CloudServices/myService/slots/Production/roles/Event.BackgroundJobsWorker.razzle/events/149d4baf-53dc-4cf4-9e29-17de37405cd9/ticks/636362258535221920",
    "eventTime" : "2017-07-21T09:24:13.522192Z",
    "contentType": "application/json",
    "schema-url": "https://github.com/Azure/azure-rest-api-specs/...",
    "data" :
    {
      ...
    }
}

Other examples

AWS Cloud Trail

Cloud Trail is similar to the service I've just dissected in detail. Here's an event
as it's in the log.

{
    "eventVersion": "1.0",
    "userIdentity": {
        "type": "IAMUser",
        "principalId": "EX_PRINCIPAL_ID",
        "arn": "arn:aws:iam::123456789012:user/Alice",
        "accountId": "123456789012",
        "accessKeyId": "EXAMPLE_KEY_ID",
        "userName": "Alice"
    },
    "eventTime": "2014-03-06T21:01:59Z",
    "eventSource": "ec2.amazonaws.com",
    "eventName": "StopInstances",
    "awsRegion": "us-east-2",
    "sourceIPAddress": "205.251.233.176",
    "userAgent": "ec2-api-tools 1.6.12.2",
    "requestParameters": {
        "instancesSet": {"items": [{"instanceId": "i-ebeaf9e2"}]},
        "force": false
    },
    "responseElements": {"instancesSet": {"items": [{
        "instanceId": "i-ebeaf9e2",
        "currentState": {
            "code": 64,
            "name": "stopping"
        },
        "previousState": {
            "code": 16,
            "name": "running"
        }
    }]}}
}

Using the same mapping approach as laid out above, I'm arriving at the following.
Mind that I am scoping the topic to EC2 in us-east-2 for the account-id 123456789012,
which is roughly equivalent to the above. For the subject, I'm choosing an expression
that captures the one instance described here.

{
    "namespace": "aws.amazon.com/awscloudtrail",
    "topic": "ec2.amazonaws.com/us-east-2/123456789012",
    "subject" : "i-ebeaf9e2",
    "eventType": "StopInstances",
    "eventId" : "b7d4398f-b2f0-4faa-9c76-e2d316a8d67f",
    "eventTime" : "2014-03-06T21:01:59Z",
    "contentType": "application/json",
    "schema-url": "https://docs.aws.amazon.com/awscloudtrail/latest/APIReference/API_Event.html",
    "data" : {
        "eventVersion": "1.0",
        "userIdentity": {
            "type": "IAMUser",
            "principalId": "EX_PRINCIPAL_ID",
            "arn": "arn:aws:iam::123456789012:user/Alice",
            "accountId": "123456789012",
            "accessKeyId": "EXAMPLE_KEY_ID",
            "userName": "Alice"
        },
        "eventTime": "2014-03-06T21:01:59Z",
        "eventSource": "ec2.amazonaws.com",
        "eventName": "StopInstances",
        "eventID": "b7d4398f-b2f0-4faa-9c76-e2d316a8d67f",
        "awsRegion": "us-east-2",
        "sourceIPAddress": "205.251.233.176",
        "userAgent": "ec2-api-tools 1.6.12.2",
        "requestParameters": {
            "instancesSet": {"items": [{"instanceId": "i-ebeaf9e2"}]},
            "force": false
        },
        "responseElements": {"instancesSet": {"items": [{
            "instanceId": "i-ebeaf9e2",
            "currentState": {
                "code": 64,
                "name": "stopping"
            },
            "previousState": {
                "code": 16,
                "name": "running"
            }
        }]}}
    }
}

Software AG Cumolocity IoT

Cumolocity is an increasingly successful IoT platform by Software AG. Here's
a simple event:

{
    "id" : "10",
    "self" : "...",
    "creationTime" : "2011-09-06T12:03:27.927+02:00",
    "type" : "com_cumulocity_model_DoorSensorEvent",
    "time" : "2011-09-06T12:03:27.845+02:00",
    "text" : "Door sensor was triggered.",
    "com_othercompany_Extension" : { ... },
    "source":{ "id":"12345", "self": "..." }
    .... data ...
}

Here's how I might map it, with a topic for each device:

{
    "namespace": "tenant.cumolocity.com",
    "topic": "12345",
    "eventType": "com_cumulocity_model_DoorSensorEvent",
    "eventId" : "a7d4398f-b2f0-4faa-9c76-e2d316a8d67f",
    "eventTime" : "2017-09-06T12:03:27.927+02:00",
    "contentType": "application/vnd.com.nsn.cumulocity.event+json;ver=0.9",
    "data" : {
        "id" : "10",
        "self" : "...",
        "creationTime" : "2011-09-06T12:03:27.927+02:00",
        "type" : "com_cumulocity_model_DoorSensorEvent",
        "time" : "2017-09-06T12:03:27.845+02:00",
        "text" : "Door sensor was triggered.",
        "com_othercompany_Extension" : { ... },
        "source":{ "id":"12345", "self": "..." }
        .... data ...
    }
}

Summary

I'm fairly optimistic that the current set of properties is already good enough
to serve the purpose of routing a dispatching events through generic middleware,
but it does take some consideration to take existing event schemas and map them
appropriately.

AWS SNS, Azure Event Grid, Google GCM and FCM, Java JMS, IBM MQ,
Apache Kafka, and MQTT (to name a few) call the top-level distribution concept
topic.

The following names are given to the accompanying next-level qualifier
that I call here subject. The subject is used to figure out in what context
a message should be processed, and for messaging aimed at humans, that's just
legible text. In some cases, "topic" where the infra-level qualification ends
and it's left to the app(s) to provide/resolve further qualification.

	Name
AWS SNS	Subject
Azure Event Grid	subject
Google GCM	title, collapse_key
Google FCM	title
Apache Kafka	n/a (no further standard qualifiers)
Java JMS	n/a (no further standard qualifiers, custom properties allowed)
IBM MQ	n/a (in IBM MQ's philosophy, topics are a taxonomy of subjects; custom properties allowed)
MQTT	n/a, see MQ, custom properties were added only in MQTT v5
AMQP	subject plus custom properties

There is very little supporting evidence in the existing product landscape across
products and services of many vendors for needing more than two standardized qualifiers
for routing an event or message.

What we do in cloud-events, is to further qualify the event for dispatching to a
handler, which then further introduces the need for eventType as a discriminator,
similar to HTTP, as an application protocol, having a notion of methods.

[duglin]

@clemensv thanks for this. While the main point of this is to help nail down the shape of the event properties, you did start out talking about the WG's scope and goals. To that end, could you look at whether you think any changes are needed to @ultrasaurus's PR (#101), and if so, suggest some edits in there. I'd like to keep all of those discussions in one spot to avoid duplicate discussion threads.

As for the properties part of the discussion, as I mentioned in a note earlier, I think a PR based on where you'd like to see us go would be a good way to help focus the discussion. If an existing PR can be reused that would be best, but since I suspect you'll be touching quite a few properties a new one will probably be needed. Just indicate which other PRs you think the new one would replace.

[yaronha]

@clemensv i'm not clear why we take fields from the event it self and copy them into the context part, in addition based on what i thought the consensus was last call that the source is the reporting entity i fail to see why you would add in the AWS case the accountId into the source field, IMO ec2.amazonaws.com is the reporting entity and to figure out the details one should peek into the event, i dont know of many protocols that replicate content into the headers in such a way, what is the benefit in doing so?

I do think having "topic" as optional field is a good idea, but IMO this is the logical Target, not the source, its not the reporting entity rather the logical end-point the event was sent to.

i also think it make sense for the EventID to be a form of a Num/Hex like in the AWS example rather than a (very) long path string, the purpose of that is to identify/track duplicates, it would be rather inefficient to hash such a long string per event

[clemensv]

@yaronha I give the reasoning for the containment strategy above. Wrapping with property promotion is a very common approach in application integration (EAI) that I apply here to those examples. When these existing log records are carried through a cloudEvents compliant infra, the existing handler code will work as-is.

I take no stance on the content of individual fields and what format they should have. That's an application concern and prescriptiveness hinders interop; at most, uniqueness is a criterion that I care about.

[duglin]

@clemensv can we close this now or is there still some action you're looking for us to take?

[duglin]

@clemensv I'm going to tag this as "GoingToClose" as I think we've moved beyond this discussion point. So, it'll have until EOD tomorrow for someone to speak-up and say they want it to remain open. ok?

[duglin]

Closing per WG agreement

[cneijenhuis]

I've worked through this to form an opinion on #406

This isn't terribly relevant, but the table says AWS SNS has a subject field, according to the definition given by @clemensv .

While SNS has a field called Subject, it does not fit the definition. SNS has made an API choice where they have some optional top-level fields that are only relevant to a specific type of subscription. E.g. PhoneNumber is only relevant for a SMS subscription. Similarly, Subject is only relevant for an Email subscription and sets the email subject. It is not typically used outside of Email subscriptions.

Should we adopt #406, I do not think the CloudEvents subject should be mapped onto the SNS Subject.

I suggest in the table, the value for SNS is changed to n/a.