Improve the catalog response format

[farshidtz]

The current API returns paginated catalog responses in paginated envelope with array of response items inside a JSON-LD object:

{
    "@context": "https://linksmart.eu/thing-directory/context.jsonld",
    "@type": "Catalog",
    "items": [ ],
    "page": 1,
    "perPage": 100,
    "total": 1000
}

The pagination attributes are passed as page and per_page query arguments defaulting to page=1 and per_page=100. The maximum allowed per_page value is set to 100 to reduce the load on server.

We need to see if this needs improvements.

Some useful relevant specs / standards:

• Collection+JSON - Hypermedia Type: http://amundsen.com/media-types/collection/
• Representing Constrained RESTful Environments (CoRE) Link Format in JSON and CBOR: https://tools.ietf.org/html/draft-ietf-core-links-json-10

Parallel discussion in WoT Discovery: w3c/wot-discovery#16

[farshidtz]

Another way to do this is by directly returning the items array (JSON Array) without the pagination object and attributes. The pagination attributes can still be responded in HTTP headers.

Example:

WordPress

Query parameters: page, per_page, offset
Example response headers:
X-WP-Total, X-WP-TotalPages

Github

Query parameters: page, per_page
Example response header:
Link: <https://api.github.com/user/repos?page=3&per_page=100>; rel="next", <https://api.github.com/user/repos?page=50&per_page=100>; rel="last"

[farshidtz]

Servers which respond with array of results inside an object:

OGC SensorThings

Query parameters: top, skip
Example response:

 {
  "value": [  ], // results
  "@iot.nextLink":
  "http://examples.org/v1.0/Things?$top=100&$skip=100"
  }

Confluence

Query parameters: start, size
Example response:

{
    "_links": {
        "base": "http://localhost:8080/confluence",
        "context": "",
        "next": "/rest/api/space/ds/content/page?limit=5&start=5",
        "self": "http://localhost:8080/confluence/rest/api/space/ds/content/page"
    },
    "limit": 5,
    "results": [  ], // results
    "size": 5,
    "start": 0
}

Elasticsearch

Query parameters: from, size
Example response:

{
  "hits" : {
    "total" : {
      "value" : 1
    },
    "hits" : [  ] // results
  }
}

OpenStack

Query parameters: limit, marker
Example response:

{
  "items": [  ], // results
  "links": [
      {
          "rel": "self",
          "href": "http://example.com/app/items?limit=30&marker=752b0b9997f24be49e5a1d89d1c53279",
      },
      {
          "rel": "first",
          "href": "http://example.com/app/items?limit=30",
      },
      {
          "rel": "prev",
          "href": "http://example.com/app/items?limit=30&marker=eff79f5b4f8743caa1f775846302c1d5",
      },
      {
          "rel": "next",
          "href": "http://example.com/app/items?limit=30&marker=08ec231f6d9a43dda97d4b950c3393df",
      },
      {
          "rel": "last",
          "href": "http://example.com/app/items?limit=30&marker=6835afb7ea29491bb2722c6c43f1f070",
      }
  ]
}

Azure Data Factory

Query parameters: limit, before, after
Example response:

{
    "data": [  ], // results
    "paging": {
        "cursors": {
            "after": "MTAxNTExOTQ1MjAwNzI5NDE=",
            "before": "NDMyNzQyODI3OTQw"
        },
        "previous": "https://graph.facebook.com/me/albums?limit=25&before=NDMyNzQyODI3OTQw",
        "next": "https://graph.facebook.com/me/albums?limit=25&after=MTAxNTExOTQ1MjAwNzI5NDE="
    }
}

CouchDB

Query parameters: skip, limit
Example response

{
    "total_rows":11,
    "offset":0,
    "rows":[  ] // results
}

Google Calendar API

Query parameters: maxResults, pageToken
Example response:

//Result contains=
"nextPageToken":"CiAKGjBpNDd2Nmp2Zml2cXRwYjBpOXA",

W3C Hyrda

Query parameters: page
Example response

{
  "@id": "http://api.example.com/an-issue/comments?page=3",
  "@type": "PagedCollection",
  "firstPage": "/an-issue/comments",
  "previousPage": "/an-issue/comments?page=2",
  "nextPage": "/an-issue/comments?page=4",
  "lastPage": "/an-issue/comments?page=498",
  "member": [ ... ]
}

[farshidtz]

We could also use the query language to navigate through resources.

E.g.:
XPath: *[position() >= 100 position() < 200]
JSONPath: $[100:200]

This even enables navigation inside individual TDs. E.g. *[id='thething']/properties[position()<5]

Edge cases to consider if we depend on query language for pagination:

1. What if no range is given?
2. What if a very large range is given?
3. How to query only the requested range from DB?

[farshidtz]

The most recent WoT Discovery spec recommends the use of arrays as payload for listing of TDs: https://w3c.github.io/wot-discovery/#exploration-directory-api-registration-listing

[farshidtz]

Implemented at /things endpoint.