Unable to upload large FileField objects using Django admin.

[rmarshall31]

When uploading a FileField object using the Django admin site, the upload fails if the file is larger than a couple MB.

Context

The error received is {"message": "Request Too Long"} if the file is ~5-10MB. Files above 10MB fail with HTTP content length exceeded 10485760 bytes. Smaller files (less than ~2MB) work fine.

I'm using django-s3-storage for static and media file storage.

Expected Behavior

The upload should succeed.

Actual Behavior

The upload fails.

Possible Fix

I believe this is a limitation of API gateway. The solution seems to be to upload files directly to S3, but I haven't found an elegant way to accomplish this using the Django admin site.

I tried zappa_file_widget, but it's out of date and doesn't work with the latest version of Django. I have to believe that I'm not the only person that has run into this and there's a better solution, otherwise I will dig into this project.

Steps to Reproduce

I have a fairly basic project at this point, the model includes a FileField. I'm using django-s3-storage. I believe the issue could be recreated using any basic project with a FileField in the model. I followed the Edgar Roman walkthrough to arrive at my current configuration.

Your Environment

I'm deploying using python 3.6 lambci docker container.

> pip freeze
argcomplete==1.9.3
boto3==1.9.116
botocore==1.12.116
certifi==2019.3.9
cfn-flip==1.1.0.post1
chardet==3.0.4
Click==7.0
defusedxml==0.5.0
Django==2.1.7
django-cleanup==3.2.0
django-s3-storage==0.12.4
django-storages==1.6.6
docutils==0.14
durationpy==0.5
future==0.16.0
hjson==3.0.1
idna==2.8
jmespath==0.9.3
kappa==0.6.0
lambda-packages==0.20.0
mysqlclient==1.4.2.post1
oauthlib==3.0.1
placebo==0.9.0
PyJWT==1.7.1
python-dateutil==2.7.3
python-slugify==1.2.4
python3-openid==3.1.0
pytz==2018.9
PyYAML==3.13
requests==2.21.0
requests-oauthlib==1.2.0
s3transfer==0.2.0
six==1.11.0
social-auth-app-django==3.1.0
social-auth-core==3.1.0
toml==0.10.0
tqdm==4.19.1
troposphere==2.4.5
Unidecode==1.0.23
urllib3==1.24.1
Werkzeug==0.15.0
wsgi-request-logger==0.4.6
zappa==0.47.1
zappa-django-utils==0.4.0

[thesunlover]

ah, I had similar problem

[thesunlover]

try to increase the AWS lambda configurations for memory & CPU

[thesunlover]

you have 30 sec for the request to succeed. this is a AWS Gateway limitation.
You can increate the timeout setting and the request will succeed, but you may not see the success in the response. if you refresh/update the page you will see that the file is uplaoded. (I was having similar problem with django csv import library which required more time to succeed)

[thesunlover]

so, the problem is not in zappa, but you may want to configure AWS lambda with zappa_settings to use more resources.
these will help you:
"memory_size":3008,
"keep_warm":true,
"timeout_seconds": 900,

I will allow myself to give some more suggestions: create a staging copy of your AWS ENV for debugging.

[rmarshall31]

Thank's for the reply. I tried increasing the memory and timeout before I posted this, and keep_warm was on by default.

Are you able to upload large files then? Which file storage module are you using?

[thesunlover]

I am using a custom version of Boto3S3Storage from the django-storages.
you can check the reason for this at jschneier/django-storages#606 (comment)
How big is your file?

[rmarshall31]

I'd like to upload files in the GB range, but right now I can't even upload a 5MB file. 1-2MB files work, but nothing larger.

[thesunlover]

then I will recommend you to use a staging ENV and allow DEBUG=True. add your email to ADMINS so that you receive the type of Exception raised.
You may want to consider the image size as a bitmap stored in RAM is less than the configured RAM.
As you know JPG is compressed, but if loaded in memmory for display or handling it may be considered as a bitmap image which takes times and times more memory as compresed.
start with debug=True, you may experience something like problem with the access or something else...

[jneves]

A lambda has limited memory and a hard limit of 500MB of storage available (and you need to account for the space the lambda is using for that). For multiple GBs you can't go through lambda.

[python1981]

Given the hard limits on filesize/timeout in ASW Gateway and Lambda, I believe the ideal solution is to:

1. Request temporary write access to a specified location in S3 which accepts arbitrarily large files
2. Instead of uploading the file in the POST/PUT, include the path to the file in S3

This ensures that timeout/filesize limits for Gateway & Lambda are not an issue because the long-running, large file upload is directly from client to S3 which is not subject to Gateway/Lambda limits.

Here is an article detailing the approach: https://sookocheff.com/post/api/uploading-large-payloads-through-api-gateway/

Zappa effectively already supports this pattern but it would require a fair bit of custom code on the backend and front-end. If anyone has ideas about how it could be streamlined in Zappa I'm sure it would be very welcome by the community.