Collections and Pagination #20
Comments
|
I like the idea of using the term Collections so that it's in keeping with an ObservationCollection. I also like the idea of having a list of collections available from the entry point. |
|
With regards to the pagination. Here's my crack at an example and we can tweak/dismiss it if required. So the user goes to the entry point: The user then follows the link to the observations collection and is presented with the following: And then if you follow the next link you'll end up with: I personally prefer the term links, as used by JSON:API, for holding the next and previous links, but view is ok if we want to stick with hydra's terminology. I've shown examples here with offset, limit and sortBy, e.g. Is it potentially a pain for end-users if we only show partial URI's e.g. Guessing we don't need to have any special HTTP headers, e.g. as described here, if we're handling the next and prev links in the JSON response? I also wonder if there's a way of preventing common share properties from being repeated. For example if all members of the collection share exactly the same |
|
My strong preference for pagination is to avoid using JSON-LD for next/prev links. The problem with this is how do you describe how to jump to a specific page, or searching of the collection. This is what I believe JSON Schema should be used for, because it has more flexibility, like defining validation on query parameters. Based IRIs shouldn't be an issue if the elements are expanded in code first, using the JSON-LD algorithms. This is something the library I've been working does automatically. |
|
So we have a And the user can look at the schema for more details on the pagination properties? E.g. what the maximum value for the |
|
Yeah, it doesn't have to be a I admit I don't know much about JSON:API though, so there might be another way. The one other thing in JSON Schema's favour though is that it is now fully aligned with OpenAPI (as of a few weeks back). |
|
Great to hear they're aligned. I'm struggling a little to see how we'll code this up in practise. Are we nearing a point where we could create a really basic Node.js application that serves some dummy observatory data using the approaches discussed? Guessing it will have the following:
This blog post introduces a few libraries that may help. |
|
Probably worth ensuring that any solution we decide upon can also handle a cursor-based approach rather than just an offset-based approach. Comparison of the two approaches here. |
|
Ok what do we think of this as an approach. A user makes the following request for observations:
To which they get the following back: Key points
This seems like a nice solution to me, although I wonder if I'm essentially replicating what JSON Schema/Hyper-Schema is supposed to achieve. |
|
As far as my experience goes, this looks like a nice solution (it's better than most data endpoints, anyway.) You've put your finger on my reservations here:
Surely it's re-inventing the wheel to invent a homebrew pagination system? Also, I don't understand where there is an |
Definitely worth avoiding this where possible. I'll raise this point on the technical call tomorrow and see what everyone thinks. Let us know if you wish to join @Joe-Heffer-Shef.
@LukeSSmith and I had a quick chat about this offline. Our conclusion being that there's use cases for either. If we do decide to adopt my approach above then there's no reason why we couldn't swap out the offset and limit properties for a cursor instead. However, we felt that when it came to requesting observations the offset, limit approach made more sense. Our worry with the cursor approach is that it could get rather complex to manage on the server/database side. The cursor approach relies on having a unique sequential column in your database table. Initially the resultTime sounds like an obvious choice for this, but then we'd get into issues when multiple observations occur at the same time. In which case do you use a sequential row index instead, but then if you do want the observations returned in chronological order, or perhaps ordered by madeBySensor then this becomes tricky. The obvious downside with the offset, limit approach is that we have streams of data coming in all the time, and thus the starting point for our offset could be changing all the time. However, the following lines in my example provide a nice solution for this: And we can always add a note in our docs tellings users to be aware of duplicates when requesting paginated observations. |
|
Yes, I'd like to attend the meeting tomorrow, please. I can see that the difficulty solving this problem arises from the same sources as many other challenges in the observatories i.e. heterogeneous data sources and unknown/varied usage patterns. |
|
Yep you've hit the nail on the head. The call is at 11:00 tomorrow on zoom. Could you send me quick message via this contact form, so I can send you the zoom details. Alternatively drop Patricio Ortiz an email as he'll be on the call too (I assume you've met). |
We need some agreement on how we manage
Collections (if we call them that... this might for example be a collection of platforms, thus paginated, as Si) that aren'tObservationCollections. Examples of how we do that would be eitherhydra:Collectionorrdf:Bag.It's also entirely possible that you might have all your platforms in one API (a lamp post API, say) and all your sensors in another (an air quality API, say) and all your historic observations in another (an observation collection API, say) and they would all just link to each other.
We also need an entrypoint that directs clients to these collections as a starting point. In other words, when I hit
https://api.example.comit gives me links to a collection of sensors, a collection of platforms, a collection of observations, etc. It wouldn't need to give me all of those necessarily, you might not have a collection of all observations from all sensors (which could be huge, but might be useful), you might only have collections of observations under each sensor.In theory, this would/could look something like...
{ "@context": { "@base": "https://api.example.com/", "uo": "https://urbanobservatory.github.io/standards/vocabulary/latest/", "title": "http://purl.org/dc/terms/title", "collections": { "@id": "uo:EntrypointCollections", "@container": "@id" } }, "collections": { "/sensors": { "@type": ["@id", "uo:Collection", "uo:SensorCollection"], "title": "All sensors available in Newcastle upon Tyne" } } }Originally posted by @LukeSSmith in #18 (comment)
The text was updated successfully, but these errors were encountered: