Streaming/chunked API (and others) response

[joetsoi]

see ckan/ckan#1847, currently the response string is fully constructed before sending it, we could do something along the lines of using an iterator to send responses so we don't end up eating large chunks of memory when sending large api responses and perhaps sending these as chunked responses?

[aliceh75]

This would be quite useful given that API consumers can be quite slow. One of our own tools streams the response from the API and processes it on the go. This means that the CKAN server has to keep the entire response in memory until the end of the processing (which in our case takes about 30 seconds for 5,000 rows).

An iterator approach with chunked responses (and/or websockets) would alleviate this.

[joetsoi]

https://stackoverflow.com/questions/6886283/how-i-can-i-lazily-read-multiple-json-objects-from-a-file-stream-in-python
https://en.wikipedia.org/wiki/JSON_Streaming

we'll have to push this to an api v4, as our output is a single json object, we'd have to redfine our output to be a list of json objects.

[wardi]

Absolutely, I'd love to see a streaming (input and output) version of our API based on newline-delimited JSON. It could be backwards compatible for old clients by requiring a prefix sent by the client like '{"streaming":true}\n'