Pagination section refers to non-existent prev_batch token when back-paginating

[richvdh]

https://spec.matrix.org/unstable/client-server-api/#pagination says 'In this scenario, the prev_batch token would be excluded from the response', but /messages doesn't have a prev_batch token. Presumably it means end.

[richvdh]

The /messages spec could do with making it clear that end will be omitted when you hit the end of the list, too.

[turt2live]

I thought we had an issue for it, but apparently not: the pagination API is inconsistently used on top of this. It's applied to the letter on some endpoints, and not at all on others. We may have to think critically about whether or not we want it, and how we want to fix it.

[richvdh]

I thought we had an issue for it, but apparently not: the pagination API is inconsistently used on top of this.

You might be thinking of #1523?

[turt2live]

Yup, that would explain why I can't find it too. I don't remember fixing it, but it sounds like I did.

[richvdh]

A list of endpoints which implement pagination:

Endpoint	Inputs	Pagination token output params	Array output param	Notes
/initialSync, /rooms/{room_id}/initialSync	limit	start, end	chunk	deprecated
/events	from	start, end	chunk	deprecated
/messages	from, to, dir, limit	start, end	chunk
/sync	since	next_batch, prev_batch	rooms/events
/publicRooms, /federation/v1/publicRooms	since, limit	next_batch, prev_batch	chunk
/keys/changes	from, to	n/a	changed, left	Uses /sync tokens
/members	at	n/a	chunk	Takes a sync token. Presumably there was an intention to add pagination later.
/notifications	from, limit	next_token	notifications	Paginates backwards from "now"
/search	next_batch	next_batch	results	Multiple levels of pagination, documented at https://spec.matrix.org/unstable/client-server-api/#pagination-1

Note that /sync and /messages have their own documentation at https://spec.matrix.org/unstable/client-server-api/#syncing.

[richvdh]

I'd like to try and come up with a convention for pagination parameters, which at least new APIs (such as the one being discussed in MSC2946, and sync v3) can follow. My thoughts so far:

• there seem to be broadly two categories at the moment: from/to/start/end/chunk, and since/next_batch/prev_batch/no chunk.
• start and end are rather generic names when used without chunk.
• On the other hand chunk seems to add overhead, and I'm not convinced it fits all use cases. It would seem a bit silly trying to use it with /sync (which has multiple pagination tokens).
• since is a funny name for APIs that paginate rather than stream.
• Other conventions in the world include before/after (Github), pagination_token->previous_token/next_token (Twitter). I like before/after but am not enthusiastic about using completely new names (and introducing an xkcd #927 situation).

Broadly I think I favour:

• input parameters called from (and to, where appropriate) for true paginated data
• an input parameter called since for streaming data
• outputs called next_batch (and prev_batch where appropriate)

As a final thought: we could do something completely different like follow Github's pattern of using Link headers to return pagination tokens rather than returning the tokens in the JSON response body. That is probably a bit too radical though.

[kegsay]

So if I understand correctly:

• Streaming /sync: accept a since input and outputs a next_batch token? "Batch" feels spurious here as there isn't really any "batches" (terminology wise) anywhere in Matrix APIs.
• Paginated /messages accept a from and to, output a next_batch (and possible prev_batch)?

This makes next_batch context-dependent, and makes it so you cannot compose streaming/pagination together without introducing a level of nesting. Sync v3 uses a level of nesting for this purpose: the next token is contained under an object called p.

These aren't problems per se, just observations based on the proposed naming. I care more about consensus more than anything else, so things like this seem reasonable.

[richvdh]

Yes, sounds like you understand correctly.

"Batch" feels spurious here as there isn't really any "batches" (terminology wise) anywhere in Matrix APIs.

Surely any API that paginates returns batches? Unless the word "batch" means something special to you that it doesn't to me? Anyway, yes I agree it's a bit redundant, but since it's reasonably widely used currently I feel like I'd rather live with that than change everything. Similarly @ara4n proposed next_token but again I'd prefer to follow the existing precedent.

This makes next_batch context-dependent and makes it so you cannot compose streaming/pagination together without introducing a level of nesting. Sync v3 uses a level of nesting for this purpose: the next token is contained under an object called p

@kegsay explained in chat that sync v3 does this to support two different types of token, where one token is used to paginate results, and the other is used to skip straight to the next point in the stream.

If this is a requirement, I think it's fine to diverge from the convention for complicated APIs like /sync which need to deal with both pagination and streaming. You could call the two different tokens next_page and next_batch, or something.

[kegsay]

FYI sync v3 now uses next_page and next_batch.