Large parallel uploads (> 100 GB) failing due to timeout?

[0sander]

Question
I am using tusd with tus-js-client to upload large files (> 100 GB) using parallel uploads (10 uploads). I see uploads failing sometimes, with the final file not being concatenated. The browser receives a timeout during the final HTTP call (Upload-Concat), which is likely caused by tusd taking too long to concatenate the file. I suspect that tusd stops the concatenation once the timeout has been reached.

Even if I set the tusd timeout to 0 (infinite) the browser will still timeout at some point - I believe it is 5 minutes in chrome.

Have you seen this behaviour with large parallel uploads and is there a solution to it? Can I make tusd send the response to the final Upload-Concat before the file is completely concatenated? I was also thinking about using onChunkComplete in tus-js-client and simply consider the upload done / resolve once all bytes have been received.

Setup details

Operating System: Linux (Ubuntu 20.04)
Used tusd data storage: disk store via NFS mount
Used tusd configuration: -upload-dir=/data/incoming -host=127.0.0.1 -behind-proxy -expose-metrics=false -hooks-dir=/data/tushooks
Used tus client library: tus-js-client using parallel uploads (10 uploads)

[Acconut]

I haven't seen this behavior before. What timeout are you exactly seeing? Is it a timeout sent from the tusd server or is it a timeout from the browser because it didn't receive the response in 5 minutes? Also, do the logs from tusd mention something?

If you are concatenation a 100GB file, you have to know that once all individual uploads are finished, tusd will read each individual upload and write the entire 100GB to disk again. Depending on your NFS setup, these 100GB might take too long to read and write, causing the long wait time. Maybe you also want to evaluate the performance of your NFS system.

Have you seen this behaviour with large parallel uploads and is there a solution to it?

No, I have never seen that before.

Can I make tusd send the response to the final Upload-Concat before the file is completely concatenated?

No, that is not possible and I don't think it would be a good solution.

I was also thinking about using onChunkComplete in tus-js-client and simply consider the upload done / resolve once all bytes have been received.

I would not recommend this approach since this will swallow errors with can occur during concatenation.

[Acconut]

Closing this issue due to a lack of information. Feel free to leave a comment if you want to continue the discussion :)