Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Memory usage with bigger files #12

Closed
deschler opened this Issue · 16 comments

5 participants

@deschler

Uploading huge files seems to be problematic. I've tried to upload a 3GB file and the apache wsgi process slowly took up the entire 12GB RAM of the server. Using the cancel link didn't stop it and i had to restart apache in order to prevent further swapping.

Now i understand that http was never designed to handle huge uploads, but i'm wondering why the process eats up that much of memory. While the file is being uploaded, the RAM usage slowly increases by the amount of the filesize, which is fine i think. After it reaches 100% upload process however, the memory usage goes up further and the file is not written.

Using the django dev server, this behaviour is even more visible. Here a file of about 200MB is enough to make my machine with 4GB RAM swap.

I override upload_complete in my custom Backend which moves the uploaded file to another location using shutil.move. I first thought this was the problem, but it's the same when i use the included LocalUploadBackend.

@skoczen
Owner

Hey,

Which backend is this with? I've uploaded 2GB files to S3 without trouble, on both dev server and live using Gunicorn.

Any more information you can provide on setup would be great.

@deschler

It's the included LocalUploadBackend, resp. a subclass of it. But it happens with both.

Setup is Python 2.6, Django-1.3.1, Apache with mod_wsgi in daemon mode on Debian Squeeze.

@imanhodjaev

The same happened today in my project served by gunicorn and nginx when tried to upload file 2 gigs

@skoczen
Owner

imanhodjaev: what backend?

deschler: Thanks for the info. Will try to reproduce here when I get time. Pull requests are always welcome if you figure it out before I do :)

@skoczen
Owner

If I had to guess on this, I'd say that it sounds like memory is being reserved for each chunk, iteratively, (5, 10, 15, 20, 25, etc) and not freed. I don't see that issue directly in the code, and haven't found time to do a deep dive.

If this is still bugging you and you have time to take a look, your ideas are appreciated!

@imanhodjaev

I guess upload issues were due to the FILE_UPLOAD_MAX_MEMORY_SIZE option.
We know Django handles uploads with MemoryUploadHandler which fallbacks to TemporaryFileUploadHandler in case if file size is out of memory. So I just set FILE_UPLOAD_MAX_MEMORY_SIZE to 350M.

@imanhodjaev

For some reason I don't know for a while AjaxFileUploader works very slowly with default upload backend (LocalUploadBackend) I just follow default tutorial described here

@davelowe

When I try using this, memory gets eaten up like bananas in a monkey pit (assuming monkeys live in pits, that is). In Activity Monitor on my Mac I watch the CPU jump up and then drop after the upload is completed, but the Real Memory number just keeps going up. Before upload, it's at 30.1MB. After uploading a 40MB file, it's now sitting at 327.6MB.

I tried a 600MB file earlier and it froze up my computer for a time. This is with the default_storage backend, with Django's MemoryFileUploadHandler in and out of the FILE_UPLOAD_HANDLERS setting, and setting to FILE_UPLOAD_MAX_MEMORY_SIZE a small and a large number.

@skoczen
Owner

Dave:

Two questions:
1) What server are you running? The built-in test server, or a proper one (gunicorn, etc)?
2) What ajax-upload backend?

My main production use of ajax-uploader is on an EC2 micro (512 MB ram), storing to s3. I'm able to upload multi-gig files, no problem, so I'd guess it's either the test env you're running (server, etc), or something in Django's localstorage backend.

@davelowe

1) Both locally on the built-in Django server and on a proper nginx server.
2) I tried two but most recently I used default_storage. I have to use the local filesystem for the project I'm using this for.

@skoczen
Owner

Hey Dave,

1) nginx - via mod_wsgi, or are you proxying to a django server?
2) default_storage falls back to django's default storage. What's django's storage backend set to?

@davelowe

Hey Steven,

1) Via mod_wsgi
2) The storage is the default

@skoczen
Owner

Hey Dave,

Thanks for the info. I haven't tested mod_wsgi (I typically proxy_pass), but I'd bet it's fine. I'll try to replicate your issue when I get a chance - as a warning, it may be a bit, things are a little crazy right now.

Thanks,
Steven

@davelowe

I appreciate it, Steven, thanks for your help!

@ericmarcos

Same problem here, with both django server and apache+mod_wsgi with local storage. Changing FILE_UPLOAD_MAX_MEMORY_SIZE doesn't make any difference. My setup is Ubuntu 12.04, Python 2.7 and Django 1.4.5.
Last comment is from a year ago, did anyone find a solution to this problem?

@skoczen
Owner

Doing a catch-up clean-out of issues on ajaxuploader. Looks like this one stalled, so I'm closing it. Please re-open if that's not right!

@skoczen skoczen closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.