Listing with pagination

[farshidtz]

This will add pagination to the spec. It includes the minimum requirements taken from #130.

This is compatible with the existing listing mechanism.

Note: This depends on #152 and should not be merged beforehand.

Todos:

• Pagination rules
• Example
• Spec in TD

Other related issues/PRs: #16, #145, #149

---

Preview | Diff

[sebastiankb]

Is there a good summary somewhere of why the earlier version of pagination was removed?

[farshidtz]

Is there a good summary somewhere of why the earlier version of pagination was removed?

I assume you are referring to #124 which was replaced by #145. There were a few issues discussed in the discovery calls and in linked issues.

I will summarize what I remember:

• The envelope wasn't valid from an ontology perspective (see Need to define the discovery types and vocabulary #54 (comment) and Handle huge set of Thing Descriptions (pagination, streaming, etc.) #16 (comment)) and we received recommendation from TD ontology authors to follow Linked Data Platform Paging 1.0. This current PR is based on it; I have to mention it somewhere.
• The pagination alone isn't sufficient to reduce load on servers/clients because the resources are of variable size.
• The envelope object around the array was inconsistent with the search APIs response format. All of jsonpath, xpath, and sparql search APIs return arrays. By returning a similar format, we simplify the specification and allow clients to receive the whole collection or a filtered subset in the same payload format. Not fully relevant, but we are now looking into merging syntactic search and listing APIs (details in Refactoring TDD Thing Description  #133).

[wiresio]

Thanks for summarizing @farshidtz !
After having read (again) through the issues mentioned and your comment I'm more confused than before:

• XPath and SPARQL queries usually do not return (JSON) arrays, see e.g.: https://www.w3.org/TR/2013/REC-sparql11-results-json-20130321/#json-result-object
• So anyway, the TDD frontend has to convert the response to another format, in our case JSON (array) for the search APIs.
• Why do we then go for JSON-LD at the listing API? Could also be plain JSON allowing more degrees of freedom.
• Phrases like "HTTP/1.1 servers SHOULD perform chunked Transfer-Encoding" are useful hints for developers but should not prescribe the individual implementation. Reducing load for servers/clients is IMHO highly dependent on the environment you're in. If a clever format, like JSON-LD / array of TDs, helps here, than we should emphasize it instead of insisting on protocol specifics

Sorry for potentially discussing the same stuff again and again, but maybe I've just missed some basic assumptions like "responses should be JSON-LD in case ... because they help / allow for ...".

[farshidtz]

• XPath and SPARQL queries usually do not return (JSON) arrays, see e.g.: https://www.w3.org/TR/2013/REC-sparql11-results-json-20130321/#json-result-object

Sorry that was a wrong statement (at least the sparql part). But for JSONPath and XPath, the result of executing the expressions to do filtering on the collection will be an array. There could be other formats on function calls for e.g. to count elements. We need to go back to the Search API spec and check if we have claimed something wrong. But that is not relevant to this PR.

• Why do we then go for JSON-LD at the listing API? Could also be plain JSON allowing more degrees of freedom.

It is (to my limited understanding of linked data), a bad practice to embed JSON-LD (i.e. an RDF document) in plain JSON. @AndreaCimminoArriaga has discussed this in the calls few times.
Note: This PR is not suggesting a new payload.

• Phrases like "HTTP/1.1 servers SHOULD perform chunked Transfer-Encoding" are useful hints for developers but should not prescribe the individual implementation. Reducing load for servers/clients is IMHO highly dependent on the environment you're in. If a clever format, like JSON-LD / array of TDs, helps here, than we should emphasize it instead of insisting on protocol specifics

That is not part of this PR. Anyway, those are recommendations based on HTTP specification to allow incremental transfer of possibly large TDs in IoT and other environments. We had a lengthy discussion in #145. The same recommendation should be added for retrieval or submission of one TD (#117). The servers are NOT mandated to follow it.

The data model, serialization format, and transfer encoding are different topics. The protocol-specific statements for HTTP/1.1 and HTTP/2 are to do efficient transfer on the named protocols.

Sorry for potentially discussing the same stuff again and again, but maybe I've just missed some basic assumptions like "responses should be JSON-LD in case ... because they help / allow for ...".

I think the reason is same as why the TD should be JSON-LD.

[wiresio]

It is (to my limited understanding of linked data), a bad practice to embed JSON-LD (i.e. an RDF document) in plain JSON. @AndreaCimminoArriaga has discussed this in the calls few times.

Thanks & apologies again for (too much) insisting. That was my missing part.

Anyway, would be great to have streamlined search APIs and even better to have both search and listing APIs streamlined.

[wiresio]

@egekorkan - I know that you are working on System Descriptions. Have you ever come across the issue of embedding TDs having their own context inside a JSON-LD frame having another context or just a plain JSON? If yes, how did you solve that? Thanks!

[egekorkan]

@wiresio so in SDs there are TD fragments that are of interest for a system and they identified in the SD ontology with the TD context: https://github.com/tum-esi/wot-system-description/blob/b45b06f97da8aca507a047865b1148d2fb3357a6/paper-mentions/3_Context/sdContext.json#L22
You can find the ontology here as well: https://github.com/tum-esi/wot-system-description/tree/master/paper-mentions/3_Context

However, I think that I do not answer your question exactly :/

[wiresio]

@egekorkan - Thanks for pointing me to your work! Looks like it is not directly applicable for TDD purposes ... was just a thought that came into my mind.