Large File upload max size

[brianmcmichael]

Cloud commander supports large file uploads, however, one of our services ( apache | nginx | passenger ) is storing these cache files somewhere under a root location in /var on websvcs08

Since /var is limited to 9.5G on the system, file uploads crash when this space is maxed out.

While uploading, we can see this space filling, but the location appears to be accessible only to root, so I will need to work with @dpjohnson to figure out where this is going.

Options:

• Increase the size of /var
• Determine which of ( apache | nginx | passenger ) is managing the upload and change the configuration to use /nfs for caching.

Disks on websvcs08

bmcmichael@websvcs08:~$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg0-lv_root
                       29G   13G   15G  48% /
tmpfs                  12G     0   12G   0% /dev/shm
/dev/sda1             969M  126M  794M  14% /boot
/dev/mapper/vg0-lv_home
                      406G   71M  385G   1% /home
/dev/mapper/vg0-lv_tmp
                      9.5G  106M  8.9G   2% /tmp
/dev/mapper/vg0-lv_var
                      9.5G  851M  8.2G  10% /var
fs10-opt.ten.osc.edu:/nfs/10/osc
                       16T  9.7T  6.4T  61% /nfs/10
fs17-opt.ten.osc.edu:/nfs/17/osc

...

Anatomy of a file upload PUT response (referencing both nginx and Passenger):

[brianmcmichael]

This appears to be system dependent. @nickjer had discussed moving the nginx temp location to the user's homedir to resolve this.

[nickjer]

Yep, will be done after Doug installs the latest nginx_stage.

[ericfranz]

@nickjer is this still a problem?

[nickjer]

Setting the nginx temp location in the user's home directory proved to be problematic. Only solution is to request from systems a larger disk where nginx stores its temp files.

[ericfranz]

Will you put this request in via service now

[ericfranz]

If we could pre-detect large file uploads we could ask users to use sftp for those and just stop them cold. Perhaps we should limit file uploads through cloud commander to 100MB file size

[ericfranz]

@nickjer hold off on that request to systems let's talk about this issue in person

[nickjer]

Procrastination pays off yet again...

[nickjer]

The maximum file size you can upload is defined by two items on the infrastructure side:

• the maximum client body size specified in nginx used for the PUN (https://github.com/OSC/nginx_stage/blob/f1377b1579e8889347d1e59491a5082d74fb90a2/templates/pun.conf.erb#L55)
• amount of available disk space in /var at the time of the upload as this is where nginx caches the client body message during an upload (https://github.com/OSC/nginx_stage/blob/f1377b1579e8889347d1e59491a5082d74fb90a2/config/nginx_stage.yml.example#L70-L72)

I am fine with setting a maximum uploaded file size limit in FileExplorer, but this will require a new issue documenting the problem and the design for this proposal. As this issue discusses infrastructure, I will close this ticket.