Streaming responses that are larger than RAM

[gitlw]

Experience Report

Note: Feature requests are judged based on user experience and modeled on Go Experience Reports. These reports should focus on the problems: they should not focus on and need not propose solutions.

What you wanted to do

As shown in https://discuss.dgraph.io/t/can-not-query-dataset-with-222m-edge-in-a-bare-metal-server-with-65gb-ram-memory-overflow/5011/6
the user issues a query to Dgraph where the expected response requires much more memory than is available on a single alpha. While we can do minor memory optimizations here and there, the current design places a limit on the size of response that a query can retrieve. It's worth at least to consider how to support responses that are much larger that the RAM.

Some potential solutions could be

• offering a streaming API for large responses
• producing a dump file on the server side, which can be downloaded by the client after it's ready

What you actually did

Why that wasn't great, with examples

Any external references to support your case

[manishrjain]

What we need to do is to optimize encoding step, because that step takes a lot of memory. @ashish-goswami is working on optimizing that.

[lgalatin]

Closing this ticket, we are tracking the progress of optimizing encoding step in this isssue: #5124