Do we need a paging functionality in bulk API?

[enikao]

Many web APIs have paging functionality, i.e. a client can limit the number of results, and then ask for all results in batches (aka pages).

Pro:

• Potentially reduces the load on the server
• If we enforce a maximum page size, we can reduce denial-of-service attacks
• Our serialization format enables paging

Con:

• We mostly send around trees, even though the nodes appear in a single list.
• The server needs to execute the (complete?) tree query to know what to return, so we might not save that much load. We might even increase the load, as the server might need to keep the query result around to reply to the next paged request.
• The client can't do much with a partial reply, as we don't know in which order the nodes appear. Until the client loads the whole reply, processing it is very cumbersome.