Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streaming responses that are larger than RAM #3934

Open
gitlw opened this issue Sep 6, 2019 · 0 comments

Comments

@gitlw
Copy link
Contributor

commented Sep 6, 2019

Experience Report

Note: Feature requests are judged based on user experience and modeled on Go Experience Reports. These reports should focus on the problems: they should not focus on and need not propose solutions.

What you wanted to do

As shown in https://discuss.dgraph.io/t/can-not-query-dataset-with-222m-edge-in-a-bare-metal-server-with-65gb-ram-memory-overflow/5011/6
the user issues a query to Dgraph where the expected response requires much more memory than is available on a single alpha. While we can do minor memory optimizations here and there, the current design places a limit on the size of response that a query can retrieve. It's worth at least to consider how to support responses that are much larger that the RAM.

Some potential solutions could be

  • offering a streaming API for large responses
  • producing a dump file on the server side, which can be downloaded by the client after it's ready

What you actually did

Why that wasn't great, with examples

Any external references to support your case

@campoy campoy changed the title Handle responses that are potentially larger than RAM Streaming responses that are larger than RAM Sep 13, 2019

@campoy campoy added this to the Future Ideas milestone Sep 13, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.