Failure to upload file of 38 GB in size

[ply112]

CKAN Version if known (or site URL)

2.6.1

Please describe the expected behaviour

Tried to upload a file (.7z format) of 38 GB in size to the file store.

Please describe the actual behaviour

At 16 minutes, the upload reaches 100% and then the website crashes and returns the following error:

"Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator at root@localhost to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log."

In the configuration file, we have ckan.max_resource_size = 500000

What steps can be taken to reproduce the issue?

It happens every time a file of this size is being uploaded.

[amercader]

Can you check the log for more details or stack trace? This will depend on how are you serving CKAN, but if you are using the default instructions it will be in /var/log/apache2/ckan.default.error.log.

This could well be an issue with configuration at the web server level. For instance if using nginx you will need to set up client_max_body_size accordingly.

In any case CKAN's filestore was not designed to handle these file sizes. You might want to consider some form of cloud storage. You can check ckanext-cloudstorage to see if it helps. The README mentions multi part uploads.

[jqnatividad]

I can confirm that ckanext-cloudstorage works quite well for large files. Even though it was built to originally target Azure, its use of libcloud allows it to be used with different cloud providers.

We tested it with AWS and we tested it with a ~50 GB file. It does multi-part uploads too, so it can do long-running uploads without timing out and messing around with apache/nginx configuration. FYI, AWS S3 can support files up to 5 TB in size.

cc @TkTech

[frafra]

This could be just the web server in front of CKAN limiting big file uploads, which is reasonable. Uploading GB sized files via HTTP forms is unreliable. Something like https://uppy.io/ should be considered instead.