Refactor pagination logic to be less complex

[LucianBuzzo]

Change-type: major
Signed-off-by: Lucian Buzzo lucian.buzzo@gmail.com

[StefKors]

Connects-to: product-os/autumndb#519

[StefKors]

Proposed Pagination API

Initial page load

1. Open a card channel with timeline with loadChannelData
2. The WS (websocket) opens a stream to the backend for each channel
3. The first message from the client to the backend is a query event and contains pagination options for each relationship type.
4. The query event registers the client's query and starts listening for changes matching the query.
5. The queryDataset event is called to request the data matching the query schema
6. The dataset gets recieved from the backend and the view is loaded.

Paginating

1. When triggering a next page load, loadMoreChannelData is called with a higher limit count.
2. This calls setSchema to update the inital query that listens to the channel to the higher limit count
3. loadMoreChannelData then calls queryDataset to trigger an event that will send the data back to the client; returning the data based on the updated query schema.

Know issues / todo's

In the PR's below I've commented and annotated what's still to be implemented and what's still not functioning 100% correctly. These are the two main aspects that haven't been implemented:

Problem 1

1. timeline: 40 messages
2. create message: 40 messages + 1 pending message = 41
3. setSchema gets called in loadMoreChannelData: 41 messages + 1 pending
4. stream recieves created message: 41 message + 0 pending

So what happens is we set the schema
and it returns the results quicker than
the new message is streamed to the client
meaning the first get an extra "older message"
before we get the new message through the

Problem 2

2 browsers side by side:

• B1 on page 2 (80 messages)
• B2 on page 1 (40 messages)

1. B1 create a message and updates it's schema to 81 messages
2. B2 will recieve the broadcasted setSchema with the limit set to 81 instead of the expected 41. See: https://github.com/product-os/jellyfish-core/blob/18c1799eee418c41d33c2f8846c0a6d3355ee81e/lib/backend/postgres/streams.js#L338-L341

Connects-to:

• Add basic support for pagination on linked cards jellyfish#5906
• Add id param to setSchema method autumndb#519