New section on Webhook message format

[mlagally]

Create a new section that defines the message format for Webhook events as a replacement for the current CloudEvents.

[egekorkan]

Before I make a PR about it, I would like to understand the requirements on the payload content for Webhook. I think that @benfrancis has more experience based on the comment at #126 (comment) . I am copying his proposal here:

• source (could be the URL of the TD, base URL, the URL of the event affordance, or the URL of the events meta interaction affordance): This is needed since the href is not necessarily unique for an event affordance in webhooks. Can be avoided if make it mandatory to have a separate href per event affordance
• subject (event name, if not provided as part of source)
• time
• id (if an identifier separate from the timestamp is desired)
• data

This is already less than what is proposed, which has fields like specversion, contenttype etc. I think we need to better understand what is needed by a Webhook Consumer. I would also prefer to omit subject and source. The presence of id is needed to better argumented. What is the motivation behind having an identification mechanism for each event emission? I can invent some but a use case would be better.

[benfrancis]

Proposal:

Member	Type	Mandatory	Comment
thingId	anyURI	Yes	The ID of the Thing from which the event was emitted.
event	string	Yes	The name of the event, taken from the key of its EventAffordance.
id	string	No	A unique identifier for the event, can be used when resuming a subscription. Can be any string including a UUID, URI or timestamp.
timestamp	dateTime	No	A timestamp for when the event took place.
data	any	No	The event payload, serialised as JSON, matching the data schema of the EventAffordance.

E.g.

{
  "thingId": "https://mythingserver.com/things/mylamp1",
  "event": "overheated",
  "id": "223e3567-e89b-13d3-a456-426653",
  "timestamp": "2021-11-16T16:53:50.817Z"
  "data": 90
}

Note that event, id and data are consistent with the HTTP SSE Profile. thingId and timestamp are new.

[mlagally]

@egekorkan

This is already less than what is proposed, which has fields like specversion, contenttype etc. I think we need to better understand what is needed by a Webhook Consumer. I would also prefer to omit subject and source. The presence of id is needed to better argumented. What is the motivation behind having an identification mechanism for each event emission? I can invent some but a use case would be better.

id's are used to distinguish and order events when they have the same timestamp.

[mlagally]

@benfrancis
I like this proposal in principle, a few comments:

• The field name "event" is rather unspecific, suggest to call it "type" with content /events/. In a similar way, property subscriptions should be /properties/
• Suggest to rename thingId to source to align terminology with CloudEvents
• Consider to include "subject", see CludEvents
• id should be mandatory, see my previous response to @egekorkan

[benfrancis]

@mlagally wrote:

Suggest to rename thingId to source to align terminology with CloudEvents
Consider to include "subject", see CludEvents

Is it more important to be consistent with CloudEvents or to be consistent with other profiles?

I thought we already agreed that being compliant with CloudEvents is not desirable, so there doesn't seem much point in partially aligning either.

id should be mandatory

IDs are not currently mandatory in the HTTP SSE Profile. They are mainly useful for catching up on missed events when re-establishing a connection, which doesn't apply in quite the same way to WebHooks.

The field name "event" is rather unspecific

I don't feel strongly about this one because I agree it isn't the most descriptive name, but it would at least be consistent with the HTTP SSE Profile.

suggest to call it "type" with content /events/. In a similar way, property subscriptions should be /properties/

That would be strange given that URL paths are not standardised in the profile. If you want to extend the WebHooks profile for observing properties you could define an alternative property member which helps differentiate property observations from events without having to use a strange naming convention in its value.

[benfrancis]

I note that defining a payload format for Webhooks has been deferred until the next version of the specification.

How should Web Things serialise event data in WoT Profile 1.0 in this case?

One very simple solution would be for Consumers to provide a different callback URL for each subscription which embeds the Thing ID and event name in the URL. That way the body of the POST request could directly contain the event data payload (serialised in JSON) without a wrapper, corresponding directly to the data schema of the EventAffordance. This would also address the kinds of concerns raised in #259 where WoT TD 1.1 Consumers which don't implement the profile don't understand the payload format because it doesn't directly map onto the data schema.

[mlagally]

@benfrancis
We should revisit what do define in the scope of Profile 1.0.

I see the following alternatives:

1. leave it open
2. Describe and recommend a single payload format - not sure if one size fits all.
3. Describe two different formats: one that is compatible with the SSE format, another one that is suitable for cloud interactions. There are clearly use cases for each of them, and these could be defined by conditional requirements depending on the deployment context

[benfrancis]

Describe two different formats: one that is compatible with the SSE format, another one that is suitable for cloud interactions.

It's not possible for WebHooks to be "compatible" with SSE since they are two completely different mechanisms, it's just a matter of whether we want some consistency in the naming of fields in the payloads across the two different profiles.

Personally I think having some consistency between profiles will make the specification easier for developers to understand when implementing it, but it's not functionally necessary since the two different event mechanisms can not directly interoperate anyway.

I thought we had already settled the debate on whether to re-use the CloudEvents specification, since it brings a lot of unnecessary redundancy of metadata which is already provided in a WoT Thing Description, and therefore unnecessarily increases the size of event payloads. I don't see any benefit of being partially consistent with CloudEvents, so we may as well define our own payload format which is internally consistent and is a better fit for the WoT data model.

Either approach would be equally suitable for cloud-based deployments, I see no difference in use cases.

For the sake of interoperability it would be preferable to define a single payload format for Webhooks.

I do think defining a payload format is necessary for out-of-the-box interoperability, but not essential for WoT Profile 1.0 if the HTTP Webhook Profile is only informative.

[benfrancis]

I note that in the W3C WebSub specification, metadata about Webhook event notifications or "distribution requests" (such as the topic URL) is provided in HTTP headers instead of in the body, so the body can be completely reserved for the event data (if any).

Instead of trying to agree on a special JSON-based payload format for Webhook event notifications (and property observations) like the one I proposed above or the one used by CloudEvents, an alternative more minimalist approach would be to dedicate the whole body of the request to the property or event data (if any), and include event metadata in HTTP headers instead. E.g.

POST https://mysubscriber.com/listeners/73d196de-db3d-4602-9bd5-7df4b7ae8ff9 HTTP/1.1
Content-type: application/json
Link: <https://mythingserver.com/things/mylamp1/events/overheated>; rel="self"
90

In the above example:

• The URL of the event endpoint is provided in an HTTP Link header with rel=self (which is how topic URLs are included in WebSub)
• The entire body (90) is the event data serialised as JSON as per the data schema of the EventAffordance (in this case it's just an integer, but could be a number, string, boolean, object, array etc.)

This is missing some information compared with my first proposal:

• The thingId is not really that important as long has you have the full URL of the event endpoint. We don't currently include the Thing ID in the HTTP SSE Profile, the Consumer just knows the event URL endpoint.
• Similarly, the event name is not really needed as long as you have the event endpoint URL.
• A unique event id is not as important for Webhooks as it is for Server-Sent Events (where they are used for catching up on missed events) because Webhooks don't rely on maintaining an open TCP socket so can just re-try delivering events until a Consumer is available again. We don't currently use the IDs for this or any purpose in the Webhook Profile. IDs are currently optional in the HTTP SSE Profile where the field is recommended to be a timestamp.
• timestamp is very useful although is currently optional in the HTTP SSE Profile. There is an HTTP Date header which could theoretically be used for this purpose in Webhooks, although that header is forbidden in the JavaScript fetch API (since it is under user agent control) so if we might choose to use a custom header instead. That would also allow us to use an RFC 3339 timestamp as used everywhere else in the WoT Profile specification, rather than RFC 5322 which is recommended for the Date header.

If any of those fields are considered essential, we could define custom HTTP headers for them, like the WebSub specification does.

An advantage of dedicating the whole body of the event notification request to the property or event data is that the data schema of the property or event can be directly mapped onto the body of the request. This could be useful for Consumers which don't conform to the Profile (and therefore wouldn't know about a special payload format), since there's no way to declaratively define a data wrapper format for event notifications (or property observations) in a Form of a Thing Description. This would also be consistent with the HTTP SSE Profile where we use the whole data field of the text/event-stream format for the event data, rather than including any kind of wrapper around it.

[benfrancis]

I think this needs addressing one way or another for Profile 1.0