Upload error on large files

[mesudip]

Describe the issue
When I upload large files, it always fails.

To Reproduce
Steps to reproduce the issue:

1. Clone repo
2. docker compose up -d
3. Upload very large file (5G+)

Expected behavior
Upload succeeds.

Screenshots

Environment (please complete the following information):

• Ubuntu
• Installation method: Docker
• Browser, if applicable

** Additional Context**
I tried increasing timeout and it's still not working

    location / {
        add_header 'Access-Control-Allow-Origin' '*';
        add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
        add_header 'Access-Control-Allow-Headers' 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range';
        add_header 'Access-Control-Expose-Headers' 'Content-Length,Content-Range';
        proxy_read_timeout 12000;
        client_body_buffer_size 50M;
        uwsgi_request_buffering off;
        proxy_buffering off;
        uwsgi_buffering off;
        include /etc/nginx/sites-enabled/uwsgi_params;
        uwsgi_pass 127.0.0.1:9000;
    }

[WaiserHai]

same problem here....

[mgogoulos]

There are two limits to set for this, 1. is UPLOAD_MAX_SIZE on settings.py, and then the client_max_body_size on deploy/docker/nginx.conf

Try them out and report here if they solve this, thanks!

[mrjoshuap]

The other scenario that plays out is possibly related to the media itself -- see #1397. Personally, I regularly upload 10GB+ media with no problem (aside from it taking a while) using the recommendations here. Otherwise, maybe it is possible you don't have enough memory for ffmpeg to transcode large media properly?

[mrpijey]

Hmm, I don't see how a misdetection of media is related? All my videos are properly detected as videos.

And there's no memory issue from what I can see, the entire docker server has 1TB of RAM and there's plenty of memory left.

Are there any logs etc I can check to see what kind of errors it outputs once it finishes uploading?

[mrjoshuap]

It's not necessarily misdetection of media; it is the "taking too long to scan for metadata" part. Just hypothesizing.

As for logs, I'm not sure; in my instance, I had to rework the logging to get useful info to debug errors like this. There is a PR for logging I submitted that greatly improves them.

[mrpijey]

OK, I reread your other post, would this metadata fix also apply to MKV files? You mentioned MP4 but I am not sure how the structure of MP4 is comparable to MKV. I could always try to do this with an MKV and see if it makes any difference.

[mrjoshuap]

@mrpijey Generally, no. MKV is a container format for multiple streams and I think the EBML (its version of MOOV) is placed at the beginning of the file; similarly with WEBM as it is MKV based. I'm not a media format expert, but I do not think the commands to move the MOOV metadata to the beginning without re-encoding would "work" on an MKV file as that metadata is supposed to be at the beginning already.

If you're getting the errors using MKV, the metadata would likely not be the culprit.

Formats that would likely be impacted would be ISOBMFF-derived formats like MP4, MOV and 3GP/3G2, as they all allow the metadata to be placed at the end.

[mbarmonde]

I decided to use Caddy vs NGINX (preference) and seemed to have fixed this issue with my setup, as I saw similar issues. I am using docker, hence leveraging local_settings.py.

Inject new nginx.conf

server {
    listen 80;
    client_max_body_size 10G;
    client_body_timeout 3600s;
    keepalive_timeout 600s;
    uwsgi_read_timeout 3600s;
    uwsgi_send_timeout 3600s;
    uwsgi_connect_timeout 300s;
    gzip on;
    access_log /var/log/nginx/mediacms.io.access.log;
    error_log  /var/log/nginx/mediacms.io.error.log  warn;

Caddy file:

    # Handle all other requests (including uploads and processing)
    handle {
        reverse_proxy http://media_cms {
            flush_interval -1
            
            # Extended timeouts for large file uploads and processing
            transport http {
                # Wait 1 hour for backend to send response
                read_timeout 3600s
                
                # Wait 1 hour to send request to backend
                write_timeout 3600s
                
                # Wait 1 hour for response headers from backend
                # CRITICAL: Allows MediaCMS to process 8GB files
                response_header_timeout 3600s
                
                # Wait 30 seconds to establish connection
                dial_timeout 30s
            }

I also added this to my local_settings.py:

# Chunked upload timeout fix
#CHUNK_UPLOAD_TIMEOUT = 7200
#UPLOAD_MAX_SIZE = 10 * 1024 * 1024 * 1024
#FILE_UPLOAD_MAX_MEMORY_SIZE = 100 * 1024 * 1024
UPLOAD_MAX_SIZE = 10 * 1024 * 1024 * 1024  # 10GB
FILE_UPLOAD_MAX_MEMORY_SIZE = 100 * 1024 * 1024  # 100MB
DATA_UPLOAD_MAX_MEMORY_SIZE = 100 * 1024 * 1024
UPLOAD_CHUNKS = True
UPLOAD_CHUNK_SIZE = 50 * 1024 * 1024  # 50MB chunks
CHUNK_UPLOAD_TIMEOUT = 7200  # 2 hours