Define protocol binding for observing properties - closes #95

[benfrancis]

This is a proposed protocol binding for observing properties in the Core Profile, as discussed in #95. It includes bindings for the following WoT operations:

• observeproperty
• unobserveproperty
• observeallproperties
• unobserveallproperties

(There are also a few minor fixes to the HTML markup for the events protocol binding which I noticed along the way).

This protocol binding uses the Server-Sent Events mechanism to push property changes to a Consumer, consistent with the events protocol binding.

Using the Server-Sent Events mechanism for observing properties isn't as obvious as using it for subscribing to events, but if a Consumer is already implementing SSE for events then it seems silly to use a completely different push mechanism for observing properties.

An alternative to Server-Sent Events would be long-polling, but this has many drawbacks:

• No built-in mechanism for re-establishing dropped connections
• No built-in mechanism for catching up on missed data
• No native API or network optimisations built into web browsers
• Higher load on the Producer/server due to repeated HTTP requests

Without this added feature currently the only way to observe property changes within the Core Profile is to manually poll a property URL with HTTP GET requests, which is very inefficient.

I would value feedback on this proposal.

---

Preview | Diff

[sebastiankb]

I think two "\n" characters are needed to end the stream.

[sebastiankb]

Maybe a JSON-based serialized data would be suitable to show here, or?

[benfrancis]

Thanks for the review.

@sebastiankb wrote:

I think two "\n" characters are needed to end the stream.

As I understand it the double carriage return comes after the last field in the message to signify the end of the message, so it's only needed after the id field in this case.

Maybe a JSON-based serialized data would be suitable to show here, or?

Technically 42 is serialised JSON :) That might be more obvious with a property with an object type, but both the properties in the example Thing used throughout the protocol binding section are simple types (a boolean and an integer) - see the Thing Description in #91. Changing that would require changing all the examples. Alternatively we could add additional more complex examples in an appendix?

[mlagally]

Maybe a JSON-based serialized data would be suitable to show here, or?

Technically 42 is serialised JSON :) That might be more obvious with a property with an object type, but both the properties in the example Thing used throughout the protocol binding section are simple types (a boolean and an integer) - see the Thing Description in #91. Changing that would require changing all the examples. Alternatively we could add additional more complex examples in an appendix?

I would encourage to think of complex properties when we design this protocol binding. Of course a single value property is trivial, and the event payload structure as well. If we delegate considerations of complex properties and cases to examples, we open the door widely for different interpretations.
For a solid specification we have to precisely define the behaviour of consumers for these cases, and I strongly support a structured event format that resolves these issues in a clear and simple way.

Please review #150 (again), which provides this kind of format. It is self contained specification text, i.e. does not have any dependency from other specifications, however is aligned with the CloudEvents spec to provide interop with that widely used spec.

[benfrancis]

@mlagally wrote:

I would encourage to think of complex properties when we design this protocol binding. Of course a single value property is trivial, and the event payload structure as well. If we delegate considerations of complex properties and cases to examples, we open the door widely for different interpretations.

There is no room for interpretation in the way that JSON is serialised and de-serialised from a string, that is defined in RFC8259.

An integer would be serialised in an event payload as follows:
data: 42\n

A more complex object would be serialised as follows:
data: {"question":"The meaning of life, the universe and everything","answer":42}\n

The purpose of a more complex example would simply be to more clearly illustrate that it is in fact JSON (which is already specified in an assertion), not to show how it should be serialised.

For a solid specification we have to precisely define the behaviour of consumers for these cases, and I strongly support a structured event format that resolves these issues in a clear and simple way.

What issues need resolving exactly?

Please review #150 (again), which provides this kind of format. It is self contained specification text, i.e. does not have any dependency from other specifications, however is aligned with the CloudEvents spec to provide interop with that widely used spec.

As already discussed in relation to event affordances (in #100, #107, #126, #134 and #150), such a data wrapper would be redundant. As far as I can tell it adds no value, all it does increase the size of the payload. What specifically do you think is missing in the payload of a property change message? What is needed except the property name, the property value and an optional timestamp?

[sebastiankb]

@benfrancis

Technically 42 is serialised JSON :)

That's right :-)

[mlagally]

The SSE event mechanism, with the underlying assumption that all connections are initiated by the consumer, blocks consumer resources (network connections) and causes protocol overhead (keep alive messages).
It is limited to use cases where a consumer (e.g. browser client) is interactacting with a few things.

It does not support battery powered devices (draining the battery due to keep alive network traffic) and does not scale to deployments with thousands of devices.

It is therefore not sufficient for critical use cases.
To enable these, we need another event mechanism, where the network connection is established by the thing only in cases, when there's an actual event.

The event payload format should be a structured JSON payload, as proposed in #150

[benfrancis]

@mlagally wrote:

The SSE event mechanism, with the underlying assumption that all connections are initiated by the consumer, blocks consumer resources (network connections) and causes protocol overhead (keep alive messages). It is limited to use cases where a consumer (e.g. browser client) is interactacting with a few things.

All event mechanisms built on HTTP, where the Consumer is an HTTP client as in the rest of the protocol binding, will share these characteristics. As discussed elsewhere, it is unreasonable to expect that all Consumers have to be both an HTTP client and an HTTP server (e.g. in the browser client case you mention where that would not be possible).

A more efficient way to listen for property changes across a large number of Things would be to share a single WebSocket connection between multiple Things (assuming those things are hosted by the same server). This is something I am exploring through the Web Thing Protocol as I also have this use case, but I think defining a WebSocket sub-protocol is probably outside of the scope of WoT Profile 1.0 at this point. I'd be happy to make support for the Web Thing Protocol part of the Core Profile, but that would significantly delay the publication timeline.

It does not support battery powered devices (draining the battery due to keep alive network traffic) and does not scale to deployments with thousands of devices.

It is very unlikely that such battery powered sensors would use HTTP if battery life is an issue. They are much more likely to use a constrained protocol over a low powered wireless network (e.g. Zigbee, Z-Wave or LoRaWAN) rather than HTTP (e.g. over Wi-Fi or Ethernet) and be bridged to the Web of Things using a gateway of some kind.

It is therefore not sufficient for critical use cases.
To enable these, we need another event mechanism, where the network connection is established by the thing only in cases, when there's an actual event.

What do you suggest? I expect you are thinking of webhooks, but as far as I know there is no way to specify a webhook connection in a Form of a PropertyAffordance in a Thing Description, since it lacks the subscription and cancellation metadata of an EventAffordance. Also, as mentioned above it's unreasonable to expect all Consumers to be both an HTTP client and an HTTP server.

I don't disagree with the limitations of Server-Sent Events, they clearly aren't appropriate for all use cases. They are however the best available option (within the publication timeline of the specification) for a deployment scenario where the Thing is an HTTP server and the Consumer is an HTTP client.

As discussed in #5 (comment), I suggest that alternative deployment scenarios (e.g. a Thing as an HTTP client with a Consumer as an HTTP server, or multiple Things hosted on a single server which can share a single connection with a Consumer) should be addressed in separate profiles, otherwise the Core Profile (or whatever we end up calling it) will become unimplementable.

[mlagally]

Profile call on April 6th:
Needs to be retargeted to SSE section.

[benfrancis]

Replaced by #191.