New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory problem downloading big resource #246

Closed
zeehio opened this Issue May 21, 2012 · 4 comments

Comments

Projects
None yet
2 participants
@zeehio
Copy link
Contributor

zeehio commented May 21, 2012

I have installed a META-SHARE node for UPC using your documentation and I am facing a problem that I am unable to solve.

Problem

I have added a big file (several GB) as a downloadable resource using storage_admin.py.
When I try to download it, I accept the licence and I click "Download". Then, the python process in the server starts to use more and more memory, until all the memory of the server is used and then it gives an error:

Unhandled Exception

An unhandled exception was thrown by the application.

Installation specifications:

  • Uses the last published release META-SHARE v2.1
  • Has all dependencies installed by install-dependencies.sh
  • Additional installed dependencies:
    • gmp-5.0.5 (I use Fedora 13, which has gmp-4 and PyCrypto complained)
    • psycopg2-2.4.2 (required by the PostgreSQL database)
    • setuptools-0.6c11(required by flup, see below)
    • flup-1.0.3 (required to have fastcgi support)
    • lighttpd (as HTTP server)
    • PostgreSQL (as database server)

Cause

Honestly, I don't know. I would think that meta-share tries to load into memory all the file that has to be downloaded, but I don't see why would meta-share do that. Could be related to the fcgi configuration? Have I done something wrong?

@cspurk

This comment has been minimized.

Copy link
Member

cspurk commented May 21, 2012

We currently only have a very basic implementation for resource downloading. So, indeed, instead of streaming a downloaded resource, we currently create an HttpResponse object containing the complete resource and only then return it; see the code.

Our long-term goal is to have lighttpd take care of all resource downloads. Before we have this, we should perhaps at least make sure that larger resources can be downloaded at all, i.e., we should implement streaming in Django.

@cspurk

This comment has been minimized.

Copy link
Member

cspurk commented May 21, 2012

@zeehio There is now a fix for this upstream. If you like, you can test it. Perhaps it will also be included in a future v2.1.x bugfix release.

@zeehio

This comment has been minimized.

Copy link
Contributor Author

zeehio commented May 22, 2012

@cspurk I have applied a513ec1 to my installation.
After a restart of the python process, I am not having this problem anymore. Thanks :-)

@cspurk

This comment has been minimized.

Copy link
Member

cspurk commented May 23, 2012

@zeehio Glad to hear this, thank you for testing and for your feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment