Handle ThingDescriptions as streams

[relu91]

In the 08/03 call, we briefly explored the idea of handling Thing Descriptions as Streams. The discussion was originally brought up by the latest advancements in WoT Profiles and WoT Discovery. In particular, this issue is linked to the idea of a minimum supported hardware platform for WoT and Handle big TDs or set of TDs.

As discussed in the call we don't have a well-defined use case extracted for those issues, but the main idea is to allow a possible consumer application to read a TD incrementally. We have pinpointed three possible places that we'll need to be updated: the getThingDescription, consume, and the discovery method.

Notice that in Scripting API the only method to fetch TDs is using the discovery method. Furthermore, if we don't want to force the implementation to store everything in memory the method getThingDescription should be modified to return a stream and the consume method too. About this last point, my conjecture is that is possible to read a TD incrementally and create a ConsumeThing without having the whole TD always stored in some memory buffer.

Still, we find that the issue needs concrete examples where the current API design fails. The optimal would be having a concrete board or hardware as an example.

[zolkis]

Question: what use case do we have for a ConsumedThing running in a constrained device?
So far, usually ConsumedThing objects have been used in a powerful device (laptop, IoT bridge/gateway/node, or cloud).

If we do have a valid use case, please add a link to this issue.

Assuming we do have a valid use case from one of the WoT member companies, coming from real deployment issues, let's discuss the further options here (theoretically, for now).

The API design should make it possible to separate the fetching, segmentation/reassembly, and parsing operations from constructing ConsumedThing. In the current API, ConsumedThing is constructed based on an init object that represents a parsed TD.

The use case is coming from the constraint that for whatever reasons, a WoT runtime can allocate only a given (say 1 KiB) size of a buffer for reading/parsing TDs on given devices.

A WoT application should still be able to construct a single ConsumedThing out of chunks of TD, i.e. sequentially (in a loop) from a number of init objects, all of which would contribute to the same ConsumedThing object. This basically means streaming input to construct ConsumedThing.

Question: it is important to know for the use case if the buffer boundaries are at stanza or byte level.

Depending on the answer, different APIs are possible.
Let's assume that buffers can split a TD at any byte. That still means that someone (WoT impl, or app) has to keep the currently being parsed stanza around in memory during parsing e.g. a Property entry, even if it uses e.g. a ReadableStream object as input.

It is safe to say that if it is not possible to parse a full stanza because buffer limitations, then there is little chance the TD could be handled by WoT Scripting, and even smaller chance that the underlying OS and HW could do anything useful via a WoT interface. These kind of restrained devices likely would need to be represented by other devices for WoT deployments.

So we could as well formulate some minimum requirements to run WoT Scripting, aligned with some real constraints, coming from real IoT/WoT deployment scenarios. That is necessary input for any change.

Until we have such requirements in place and analyzed, I think it is okay to assume that the minimum requirement for WoT Scripting is that the object resulting from parsing a TD can fit in the device memory.
Until then we can stay with the current API. The question is that if a change will come, will it need to break the current API.

If the minimum requirement is that at least a full stanza (i.e. any property, action, event definitions) can fit in the memory, then I would go with the current API + one more method, for extending a ConsumedThing object with another init object.

If a TD is served by chunks (stanzas), every stanza (interaction description) that fits in the memory could be represented by ConsumedThing. Only the interaction descriptions that fail to fit in the memory will fail to be added to ConsumedThing, hence all operations related to that interaction will fail. I think that would be still better than failing the whole construction, but this could be an app option.

This would cover this use case, and even more. A similar proposal for ExposedThing has been proposed here.

partial interface ConsumedThing {
    Promise<undefined> extendFrom(ThingDescription input, optional ExtensionOptions options);
};

enum MergeOptions { 
    "strict", // error on every stanza conflict, i.e. only additions
    "override", // input replaces conflicting stanzas
    "try", // on conflict, keep the old items, i.e. add but don't break
};

dictionary ExtensionOptions {
  required MergeOptions mergeOptions = "strict";
  required boolean allowFailingStanzas = true;
};

Notes:

• ThingDescription (defined as object) would then mean either a full TD or a full stanza (e.g. a single Property definition).
• There is no need to check by the implementation if the chunks are part of the same TD or not: the app (doing the fetch) should be in control.
• Empty ConsumedThing constructor should then be also allowed (with all methods failing, of course).

Compatibility with the current API would be good:

• there is only one new method, which can be even placed in a separate conformance class
• most new stuff can be encapsulated by implementation
• what might change is that empty ConsumedThing constructor MAY be allowed in the future. This will not break existing code.

Another corollary is eventually changed behavior for getThingDescription(). The possibilities are the following:

1. Current behavior: if a TD cannot be represented by an object that fits in the memory, fail this method.
2. Return a ReadableStream object that will provide a TD. However, I think it is extremely unlikely that a given HW will be able to host a JS runtime that supports the huge streaming objects, but won't be able to support a parsed object that represents a TD that the device can handle.

Summary

• I don't think that using e.g. a ReadableStream as input would solve a real use case (because of the requirements/constraints scissor as described above is closed).
• We can make small changes to the API that would support chunked construction of a ConsumedThing, and if done in the way suggested above, it might be also useful for other use cases. But the added complexity means we have to carefully consider the real use cases, exactly as @relu91 has stated above.