Add request limit and offset query parameters

[calebmer]

Once I saw the way PostgREST handled pagination, I was ready to leave the traditional limit and page query parameters behind. However, @davidthewatson (when talking about something else entirely) brought up the HATEOAS argument for query parameters to control pagination. In addition, there is currently no mechanism for paginating embedded collections.

This PR aims to solve these two problems:

1. HATEOAS pagination.
2. Collection embedding limiting.

The way this PR accomplishes this is by adding three query parameters: limit, offset, and page (if offset is not defined and page is, limit * page = offset). This PR also allows these parameters to be qualified, for example posts.limit=4 to limit embedded collections. These parameters do not replace the range headers, but rather supplement. Extending the existing mechanisms.

I was going to wait until I had finished the collection limiting before submitting this PR, however @begriffs in gitter alluded to a large rewrite as he upgraded to the newest Hasql. Therefore I really don't want to go too deep into this PR before that upgrade has been merged. For now the only new features are the global request headers.

Also, I haven't currently implemented Link headers for pagination because I'm not exactly sure how they would work. I'm thinking they would only be included if a page query parameter was specified.

[begriffs]

Thanks for all the contributions lately!

I remember thinking about hateoas in 2014 and concluding that range headers ought to supersede pagination link headers. Would you mind reading my short article about it and telling me what I might have overlooked? http://begriffs.com/posts/2014-03-06-beyond-http-header-links.html

[calebmer]

I completely agree with the reasoning, but there's a couple of reasons why I still think the query parameters are useful.

First is for logic duplication. In advanced scenarios random access is super convenient for the developer. However, in simple ones the developer has to store more state and write more logic. It's much simpler to develop a pagination client which just looks for link headers.

Second is for human access. I've found it's hard for my brain to think in ranges. When I want to get the first two things I'll write 0-2 which is of course, wrong. It's easier for a human to think in terms of limits and offsets when specifying a range. I think that's why SQL did it.

Third is for recursion. We need a way to paginate embedded collections, this gives us a mechanism.

I completely agree that the headers implementation is more flexible and I don't want it to ever go away. But for various reasons I think it's helpful to have the headers enhanced by the query parameters.

[davidthewatson]

I agree with most things said here on both sides which probably indicates some ambivalence on my part.

¯_(ツ)_/¯

My only additional observation is that, while I agree with @begriffs reasoning in the linked article, from my experience, the HATEOAS style advocated by @calebmer is more common, while noting the sample size n=1.

When I ran into this problem coding the PostgREST client, I went and read the HTTP 1.1 spec and while it does explain the semantics, it could be confusing to the uninitiated.

The term affordance is certainly appropriate in describing the solution. We essentially want to grease the path toward wide client adoption, without compromising the integrity of the server implementation.

Pagination usability is certainly a problem related to mental models of pagination state. Having client examples certainly helps, but link alternatives would be a natural fit for those who have ridden the HATEOAS bus. And while I'm not a visual person, a graphic might help, though I'm incapable of rendering it myself.

You essentially want to A/B test the two pagination styles or perhaps do a survey?

[ruslantalpa]

I just glanced over the code but am i right when i say that the "qualify" part only works one level, i.e. posts.limit works but posts.comments.limit does not?

Short version, +1 on the idea/direction -1 on the implementation :)

About the implementation:
The data types used in the AST should describe the limit/offset feature and each node of the tree should have those properties (the same goes for ordering while we are at it). The parsing of the limit/order parameter(s) should be done at the parsers stage.

About the idea:
While i personally do not immediately see the usefulness of being able to paginate the embedded collections, there are probably many use-cases for limiting the result set. Also being able to order the embedded stuff if probably very useful. I look at limiting/ordering as two features that are very similar when it comes to how they are implemented and how they are expressed in a http request so try to think about both of them at the same time.
The Range header is a great idea and it worked well while the thing being returned was "flat", but now i don't see how one can express with range headers the need to limit stuff for each node in the tree. There is also be this "disconnect" that the limit is expressed in a header and order as a parameter.

I propose that we switch to expressing order/limit/offset in GET parameters while also maintaining backward compatibility for expressing the limit with ranges for the root node of the response, like so

&limit=10&offset=20 --for the top node
&posts.comments.limit=10 -- for a deeper node
&order=.... -- for the top node
&posts.comments.order=.. -- for a deeper node

The semantics of the request http request start to resemble SQL (select,limit,offset,order) so i don't think "page" support should be included, it can be modeled by the (js) client.

[calebmer]

Sorry for not responding, I haven't had a lot of time to work on this, and (no rush) but I am waiting on the hasql upgrade to land so I don't have to deal with too many merge conflicts.

@ruslantalpa the qualify part was not well thought out and is currently not used 😊. As the code stands it doesn't have the posts.comments.limit. So I agree, the implementation does need work. That's a good point about ordering (and I might even prefer extending the idea to select so embedding is done via &posts.select=id,title vs &select=posts{id,title}), that shouldn't be too hard to do.

As for a page parameter, I get your point about the SQL operators but I like page for HATEOAS reasons alone. It's super nice to have link headers where page is set. On that note, maybe page should only be applicable at the root level?

@davidthewatson I would really like an A/B test or survey mechanism for PostgREST, but I'm not sure (maybe @begriffs knows more) of a way to get in touch with all of PostgREST's users. I also agree with your affordance point.

[begriffs]

Hasql upgrade is done and merged now.

[calebmer]

I won't be able to work on this for a while, so I'm going to close and hopefully reopen when I get some time.