Add Discovery section to HTTP Baseline Profile - closes #112

[benfrancis]

This PR adds a Discovery section to the Core Profile as suggested in #112.

I propose that the only mandatory discovery mechanism for the Core Profile should be Direct discovery, simply meaning that the Thing's Thing Description must be retrievable from an HTTP URL.

---

Preview | Diff

---

Preview | Diff

[benfrancis]

In the last call it was proposed that Well-Known URI discovery mechanism also be made mandatory in the Core Profile.

I have spent some time looking into this further and wanted to report back my findings.

tl;dr: My recommendation is not to make the well-known URI discovery mechanism mandatory for the Core Profile because in many cases it also has the unintended consequence of making the Directory Service API mandatory too.

As I think @danielpeintner mentioned on the call, there's a limitation which means that there can only be one well-known URI per origin, at /.well-known/wot-thing-description. For origins which only host one Thing this is fine, but for cases where multiple Things share the same origin this URL must return the Directory Description of a Thing Description Directory.

This means that a consequence of making the well-known URI discovery mechanism mandatory would be that for implementations which host multiple Things (e.g. a gateway or directory or just a device which exposes itself as two Web Things), it would also be mandatory to provide a Directory Description and implement the whole of the Directory Service API as well.

I don't necessarily think it's a bad thing for the Core Profile to make the Directory Service API mandatory, but it would significantly increase the amount of work needed to make implementations conformant with the Core Profile (this is particularly onerous as long as the write and search functions are still mandatory, though I'm hoping those requirements might be dropped, see w3c/wot-discovery#208).

In the short term this may also cause a problem with making sure we have two implementations of the specification for standardisation purposes. Whilst it is my intention to implement the Core Profile on WebThings Gateway over the next few months as the specification stabilises, although I would like to I can't guarantee implementing the Directory Service API as well since I don't currently have enough funding for that. If there isn't another implementation within the publication timeline of the Profile specification then this might put this feature at risk.

I don't imagine that when people suggested making the well-known URI discovery mechanism mandatory it was also their intention to make the Directory Service API mandatory too, so I think this warrants further discussion.

I also don't particularly think that the well-known URI discovery mechanism adds much value over the Direct discovery mechanism. For the Direct Discovery mechanism a Consumer needs to be provided with the URL of a Thing Description (e.g. https://mythings.com/things/foo), but for the well-known URI discovery mechanism it still needs to be provided with the host (e.g. https://mythings.com) in order to derive the URL of the Thing Description (e.g. https://mythings.com/.well-known/wot-thing-description). In some cases the Thing Description URL and host URL may even be the same (e.g. https://toaster42.things.com/). The well-known URI discovery mechanism also puts an additional constraint on the URL of a Thing Description since it always has to have an absolute path of .well-known/wot-thing-description and can't just be hosted at the root domain for example (unless a redirect is implemented).

Taking into account all of these factors my recommendation is not to make the well-known URI discovery mechanism mandatory for the Core Profile.

There may be an argument for making the Directory Service API mandatory in cases where a single origin hosts multiple Thing Descriptions, but that's really a separate discussion.

CC @mlagally, @mmccool, @sebastiankb, @farshidtz, @relu91, @egekorkan

[mlagally]

I think requiring the directory API is not what we want to achieve here.
I support the self-descriptive approach and suggest that the security TF considers to isolate the well-know URI
discovery, which then could be adopted by the profile.

[benfrancis]

@mlagally wrote:

I think requiring the directory API is not what we want to achieve here.

I agree that wasn't the intention, and that would be a separate discussion.

I support the self-descriptive approach and suggest that the security TF considers to isolate the well-know URI
discovery, which then could be adopted by the profile.

Putting the issue of the Directory Service API aside, I still don't think it's possible to make the Well-Known URI discovery mechanism mandatory for all Web Things conforming to the Core Profile. You can only have one well-known URI per origin, so it doesn't work for cases where multiple Thing Descriptions are hosted from the same origin.

I don't think this is something that the discovery task force (I assume you meant them, and not the security task force?) can fix, because it's inherent in the way well-known URIs work (see RFC8615).

My recommendation is therefore that only the Direct discovery mechanism (i.e. that a Thing Description must be retrievable from an HTTP URL) should be mandatory for the Core Profile, so we should land this PR as it is.

[mmccool]

For reasons other than those cited (which I think are all fixable) I would not recommend making .well-known mandatory since it's only useful for self-description, and making self-description mandatory has privacy* implications. Note however it is mostly for the case where you don't have the real URL, just e.g. an IP address.

The privacy implications are mostly because of the frustrating lack of a general security/authentication mechanism that can be supported in all use cases, esp LANs. Depending on "closed networks" is in fact current practice in many home IoT deployments but is somewhat... unsatisfying.

[mlagally]

@mmccool

For reasons other than those cited (which I think are all fixable) I would not recommend making .well-known mandatory since it's only useful for self-description, and making self-description mandatory has privacy* implications. Note however it is mostly for the case where you don't have the real URL, just e.g. an IP address.

I think the privacy problem can be solved even with a mandatory self description mechanism.
There are two cases:

• If the caller is trusted by the device (or no privacy sensitive information is kept in the device) the well known URL reorts full data.
• in the other cases, it just reports the kind of information that can be freely exposed to anyone. In some cases this may be very limited content or even no content at all.

With these thoughts, I suggest to make the well known URL mechanism mandatory.

[mlagally]

As of today, the current discovery specification is in flux and may create significant additional implementation burden on the consumer.
See w3c/wot-discovery#272 for details.

I suggest we hold off with this MR until the discovery spec is consolidated.

After that we should consider whether the constraints that are defined in the discovery specification are causing significant implementation burden.

[mlagally]

Suggest we wait until the discovery spec is more stable.

[benfrancis]

Proposal based on w3c/wot-discovery#263 (comment)...

How about we say that for the HTTP Baseline Profile:

1. Conformant Things MUST have a Thing Description which can be fetched from an HTTP URL
2. Conformant Consumers MUST be able to fetch a Thing Description from an HTTP URL

In other words, Things and Consumers must support the Direct introduction mechanism and Self-description exploration mechanism, but don't necessarily have to implement Thing Directories and Thing Links.

[mlagally]

Arch call on 11.5.:
PR has merge conflicts.
It puts a mandatory requirement on system behavior that is too narrow, i.e. prevents out of band communication of a TD (e.g. via email)
We could add a link to the TD with a URL where the TD is hosted, this could also be a self descriptive URL.

[benfrancis]

Yes the PR needs some work, but I was hoping to agree on the general requirement first.

It puts a mandatory requirement on system behavior that is too narrow, i.e. prevents out of band communication of a TD (e.g. via email)

It doesn't prevent the TD being shared by email, it just requires that TDs which conform to the HTTP Baseline profile can be fetched over HTTP. That means that any conforming consumer will be able to discover any conforming Thing using the Direct introduction mechanism. If no discovery mechanisms are mandatory then the TD may not be discoverable at all by conforming consumers, and therefore not interoperable.

We could add a link to the TD with a URL where the TD is hosted, this could also be a self descriptive URL.

I see now harm in a self link in the TD, but how does that help with discovery?

[benfrancis]

It's currently looking like support for "security bootstrapping" (aka "Thing Description authentication") may be made optional in the WoT Discovery specification. This may mean that out-of-the box interoperability between WoT Consumers and WoT Directories (or in fact self-describing Things) is not possible for Thing Descriptions which require authentication.

If this is the case, I think it further strengthens the argument to add a Discovery section to the HTTP Profile which adds additional constraints on top of what is described in the WoT Discovery specification. Specifically:

1. Conformant Things MUST have a Thing Description which can be fetched from an HTTP URL
2. Conformant Consumers MUST be able to fetch a Thing Description from an HTTP URL
3. Conformant Things, Directories and Consumers MUST support a finite list of authentication mechanisms for accessing Thing Descriptions over HTTP

Without this third constraint it's likely that (where TD authentication is required) only WoT Consumers and WoT Directories (and Things) from the same vendor will work together, since Consumers will require custom code for each Directory implementation they consume.

See also: w3c/wot-discovery#313 (comment).

[mmccool]

I am not against some additional constraints on Discovery in the http-baseline-profile but want to make some observations:

• Support for Discovery itself is optional (SHOULD) in the WoT Architecture document. Making it mandatory (subject to some constraints) would help interoperability, certainly.
• The "Direct" introduction method does not deliver TDs, it is an "Introduction mechanism" and is just a means of providing a URL that could also point at a Directory (for example). The PR as written seems to combine the concepts of the Direct Introduction and a TD Server (which actually is the service that returns a TD when given a URL from an Introduction mechanism). Also "Direct" is kind of a catch-all for "other" sources of URLs and is not really the right thing for improving interop, since we still don't know where the URL will come from. What Discovery does require is that at least one introduction is supported.
• I think it is reasonable to insist that Things conforming to the profile support both the well-known and direct introduction mechanisms. Earlier I had concerns about privacy with well-known, but that can be mitigated by supporting authentication (perhaps with security bootstrapping). In practice what you would want to do is use security bootstrapping just for the initial connection of a Thing to a hub, and then (once you have established better security, e.g. installing PSK) disabling the security bootstrapping. In other words, a onboarding process. Really this process should be a whole other deliverable (and I plan to propose we work on such a deliverable in our next charter) but we could perhaps lay the groundwork for it. Once you have set up a "group key" for accessing TDs at least, then the .well-known URL is still useful for all the devices using that key.
• Two things we could do in addition for profiles are: 1. providing a common way to disable security bootstrapping 2. providing a common way to establish a group key for accessing TDs via .well-known. Steps 1 and 2 could be combined in a single standardized action. It might be a good idea to also define a common "offboarding" action that removes these keys and returns it to its original state, and a side-effect of this should also be to delete any other configuration data.
• We did some work to simplify Directories to be able to support a static listings, e.g. for the use case of a single device hosting a fixed (small) number of Things. Only the "listing" functionality is mandatory, now. I think there are use cases even for small devices where this is useful (e.g. multi-sensors). I did an implementation of a simple directory with only the required functionality and it does not require that much code, although it does require HTTP (but this is not a problem for the HTTP Baseline Profile).

[mmccool]

I think with a few tweaks, to mention a TD Server that the URL provided by Direct points at (a Direct Introduction itself does not provide a TD, it points at a TD Server that provides it) this could be merged. But the whole problem does need more thought.

[mmccool]

With the new terminology, what you want to say here is the Thing Description for a Thing must be available from a TD Server, given a URL provided with (at least) Direct introduction (which just means the user needs to provide the URL, somehow).

See my other comments. Given we are talking greenfield devices, it might be best after all if we require .well-known and self-description, as long as we can resolve the privacy/security issues.

[mmccool]

maybe "retrieve a TD from a TD Server" using the new terminology in WoT Discovery.

[mlagally]

@benfrancis

It's currently looking like support for "security bootstrapping" (aka "Thing Description authentication") may be made optional in the WoT Discovery specification. This may mean that out-of-the box interoperability between WoT Consumers and WoT Directories (or in fact self-describing Things) is not possible for Thing Descriptions which require authentication.

We could (and should) use a self descriptive mechanism, which may require authentication. This would be up to the device manufacturer to decide.
For devices without sensitive TDs, the well known URI returns the full TD. For sensitive devices, if the consumer does not authenticate, the well known URI just returns a crippled TD that only contains info about the supported auth schemes. After successful authentication of a consumer, it would return the full TD.

All OOTBI scenarios would work, if the consumer knows the credentials.

If this is the case, I think it further strengthens the argument to add a Discovery section to the HTTP Profile which adds additional constraints on top of what is described in the WoT Discovery specification. Specifically:

1. Conformant Things MUST have a Thing Description which can be fetched from an HTTP URL
   This requirement seems a bit restrictive and needs some clarification. What is the purpose of this constraint? What if somebody provides a TD over a different protocol? The Thing would be completely usable and all OOTB use cases would work. What if somebody uploads a TD to a server in an HTML form?

2. Conformant Consumers MUST be able to fetch a Thing Description from an HTTP URL
   Same concerns as on initial version #1.

3. Conformant Things, Directories and Consumers MUST support a finite list of authentication mechanisms for accessing Thing Descriptions over HTTP

I'm ok with the finite set of auth mechanisms, we may be even able to define the list.
However so far the profile does not talk about directories at all and I don't think the profile should mandate them. The deployment scenario / business case for directories needs to be clarified.

[mlagally]

@mmccool

I am not against some additional constraints on Discovery in the http-baseline-profile but want to make some observations:

• Support for Discovery itself is optional (SHOULD) in the WoT Architecture document. Making it mandatory (subject to some constraints) would help interoperability, certainly.

I think the profile should not make directories mandatory, because profile compliant things will work in many use cases that do not require them.

• I think it is reasonable to insist that Things conforming to the profile support both the well-known and direct introduction mechanisms. Earlier I had concerns about privacy with well-known, but that can be mitigated by supporting authentication (perhaps with security bootstrapping). In practice what you would want to do is use security bootstrapping just for the initial connection of a Thing to a hub, and then (once you have established better security, e.g. installing PSK) disabling the security bootstrapping. In other words, a onboarding process. Really this process should be a whole other deliverable (and I plan to propose we work on such a deliverable in our next charter) but we could perhaps lay the groundwork for it. Once you have set up a "group key" for accessing TDs at least, then the .well-known URL is still useful for all the devices using that key.

This is similar to the approach in my previous comment, bnut it needs more thought. Who would provide this "group key"? Can it be revoked/updated when a device is transferred to a new owner?

[mlagally]

@benfrancis
please change "core" to "baseline"

[mmccool]

@mmccool

I am not against some additional constraints on Discovery in the http-baseline-profile but want to make some observations:

• Support for Discovery itself is optional (SHOULD) in the WoT Architecture document. Making it mandatory (subject to some constraints) would help interoperability, certainly.

I think the profile should not make directories mandatory, because profile compliant things will work in many use cases that do not require them.

I said making Discovery mandatory, not Directories...

[mlagally]

Arch call on July 20th:
It seems to ok to require that TDs have a HTTP URL, wording of the spec needs to be aligned with the discovery spec.
Proposed way forward: upate language, resolve merge conflict.

[benfrancis]

OK, I've reduced this down to the simplest, least contentious form I can think of, following the new terminology in the WoT Discovery specification.

In order to conform with the HTTP Baseline Profile, a Web Thing's Thing Description [wot-thing-description] MUST be retrievable from a Thing Description Server [wot-architecture11] using an HTTP [HTTP11] URL provided by a Direct Introduction Mechanism [wot-discovery].

Hopefully this establishes the most basic requirement for discovery via HTTP.

---

I think two further points require further discussion as follow-up work:

1. Whether to make some form of security bootstrapping mandatory
2. Whether to make the well-known introduction mechanism mandatory

I would like to do the former if possible, but it will require defining a fixed set of authentication mechanisms (see #250). I don't think the latter is necessarily a good idea, because of the limitations of only one well-known URL per origin.

[benfrancis]

@mlagally wrote:

For devices without sensitive TDs, the well known URI returns the full TD. For sensitive devices, if the consumer does not authenticate, the well known URI just returns a crippled TD that only contains info about the supported auth schemes. After successful authentication of a consumer, it would return the full TD.

I originally thought this was a good idea too, but we've now established that this doesn't work as a bootstrapping mechanism because the security metadata in a Thing Description refers to the endpoints of interaction affordances, not to the Thing Description resource itself. See w3c/wot-discovery#135.

The WoT Discovery specification now describes another mechanism in 7.1.2 Security Bootstrapping which uses HTTP 401, 302 or 303 response codes (depending on the security mechansim being used). Support for this bootstrapping mechanism is not currently mandatory for either Producers or Consumers, so as things stand only unauthenticated Thing Descriptions (consumed by Consumers which support HTTP and a Direct introduction mechanism) are guaranteed to be discoverable with the above addition to the HTTP Baseline Profile.

[mlagally]

@benfrancis Thanks for fixing the merge conflict.
The discovery section targets the baseline profile only, so strictly speaking TDs that conform with the SSE or WebHook profile won't be discoverable.
I suggest you move the proposed text to chapter 6, which should contain all shared mechanisms and constraints.

[mlagally]

I think this language is problematic, since it mandates the presence of a TD server as part of a deployment of profile compliant devices and I don't understand the rationale behind it.

How does the manufacturer of a thing ensure that the TD is deployed in this way? What if all TDs are stored on an old-fashioned FTP server or are stored in the file system (or hard coded) in the consumer?

I don't see a clear business model for a 3rd party to provide a TD server with a revenue stream that pays for running the server.
I'm afraid that if we enforce that constraint, we might risk adoption of the profile.

[egekorkan]

I sort of agree with this comment but we don´t have to make a one size fits all profile anyways, just an implementable subset is fine. If we are trying to make something that fits all, lots of other parts of the spec are open for change.

[benfrancis]

@mlagally wrote:

I think this language is problematic, since it mandates the presence of a TD server as part of a deployment of profile compliant devices and I don't understand the rationale behind it.

A TD server is just an HTTP server which can serve a single JSON Thing Description. A Web Thing conforming to an HTTP already has to have an HTTP server serving JSON resources, so there's no reason the Thing Description (the most important resource of all) couldn't be served over HTTP too.

How does the manufacturer of a thing ensure that the TD is deployed in this way?

The obvious way is for the Web Thing to serve its own Thing Description, as described above. But they could host it elsewhere if they really wanted to... I don't think I understand the problem.

What if all TDs are stored on an old-fashioned FTP server or are stored in the file system (or hard coded) in the consumer?

Then it's not compliant with the profile. I don't know why on earth a manufacturer of a greenfield HTTP-based device would want to do either of those things. Hard-coding TDs in a Consumer is particularly counterproductive as far as interoperability is concerned.

I don't see a clear business model for a 3rd party to provide a TD server with a revenue stream that pays for running the server.

They don't need to. The Web Thing can serve the TD itself.

I'm afraid that if we enforce that constraint, we might risk adoption of the profile.

I'm afraid that if Thing Descriptions don't have even one single basic mechanism for discovering them, it will risk adoption of the profile. Do you have another proposed mechanism other than the HTTP profile requiring the TD be served over HTTP?

I'm really surprised that having the HTTP Profile require that Thing Descriptions are served over HTTP is in any way controversial. I thought we already agreed that in the last meeting:

Arch call on July 20th:
It seems to ok to require that TDs have a HTTP URL, wording of the spec needs to be aligned with the discovery spec.
Proposed way forward: upate language, resolve merge conflict.

[benfrancis]

@mlagally wrote:

I suggest you move the proposed text to chapter 6, which should contain all shared mechanisms and constraints.

OK, let's do that in #214.

[mlagally]

Arch call on Aug 3rd: agree to merge