[Feature Request] Extend API to enable pagination in UI

[bartvanb]

Problem description
We would like to use pagination in the UI to make everything go a lot faster, but we're missing some things on the API to make that change. This should be included in v4.

Features that we definitely need:

• some type of include argument where you include complete resources. E.g. /node?include=collaboration,organization should yield the node with inside the full json of the organization and collaboration. (Note that this could lead to difficult authorization checks)
• A count option to get the total number of resources without needing to get them all fully

We should consider if these are all necessary features, or if there are more

[bartvanb]

With the include argument, I think it should be possible to do the authorization checks in a central place (maybe with a few variations)

[bartvanb]

After rethinking this, we want to go a different way

• No include arguments as this leads to complicated permission checks and potentially unlimited nestings
• The count option is already there: it's just 'hidden' in the headers

Instead, we should:

• Adapt the API so that everything is always paginated
• Adapt the UI to go through all the pages whenever necessary

And also

• we should have a sort parameter so that when sorting tables in the UI, we can always get the top 10
• The sort should also contain options to sort ascending or descending
• There should be an error if people ask a page higher than is available
• Errors for wrong pagination parameters should be visible to user
• Ensure that run.list() and run.list_open() functions in NodeClient are returning all results (or are called iteratively)
• Check AlgorithmClient for paginated results
• no_pagination option on /rule` endpoint because it is an often-used endpoint that will never return large data
• user_id filter option on /rule endpoint to conveniently get all rules for a user in 1 API call

[bartvanb]

Relevant branch: change/always-pagination
For UI: feature/pagination

[frankcorneliusmartin]

🔴

From the client:

>>> len(client.organization.list(per_page=2)['data'])
Making request: GET | http://localhost:5000/api/organization | {'page': None, 'per_page': 2, 'name': None, 'country': None, 'collaboration_id': None}
Starting new HTTP connection (1): localhost:5000
http://localhost:5000 "GET /api/organization?per_page=2 HTTP/1.1" 200 5562
6

A more basic requests yields the same:

>>> len(client.request('organization', params={'per_page':2})['data'])
Making request: GET | http://localhost:5000/api/organization | {'per_page': 2}
Starting new HTTP connection (1): localhost:5000
http://localhost:5000 "GET /api/organization?per_page=2 HTTP/1.1" 200 5562
6

Version: 4.0.0a7

[frankcorneliusmartin]

Interestingly enough this does work for the client.node.list(.) function / endpoint