Upload error for large files (Warning: Request Entity Too Large (413))

[khabatv]

used to to be possible to upload the data in the galaxy (https://usegalaxy.eu/) without any problem. but now it gives an error about the size.
this is the case for the files bigger than 1gb size.
Warning: Request Entity Too Large (413)Describe the bug
A clear and concise description of what the bug is.

Galaxy Version:
https://usegalaxy.eu/ at 06.10.2021

To Reproduce
Steps to reproduce the behavior:

1. Go to "Upload Data"
2. Click on ' choose a local file and select a file larger than 1gb'
3. Scroll down to 'Start'
4. See error

Expected behavior
uploading the data normally

Screenshots

itional context**

[hexylena]

@gmauro, in case you missed it

[gmauro]

Thank @hexylena
@khabatv Uploads on website are limited to maximum 1GB file size. For larger files, you can use our ftp service.
All the detail to use the service at https://galaxyproject.eu/ftp/

[hexylena]

we should reopen as a galaxy issue, if the FTP server is configured, it should give the additional hint of "try ftp" and link to that, like we do in other places.

[mvdbeek]

FWIW I'm working on overhauling the chunked uploader so it will support resume and various backend solutions via https://tus.io/. But that'll probably only be in the 22.01 release.

[hexylena]

I saw that mentioned! It'd be great, but I think the chunked uploader had issues unrelated to its implementation for us on EU and similar solutions I'm not sure will fix that. (Namely chunked content was buffered once by nginx to disk, once by uwsgi, and then finally reassembled into a file, which was hard on our FS)

[mvdbeek]

That's mostly (or all ?) solved. The old chunked uploader did not use any of the available optimizations and was very naive. There's still going to be re-assembly of the file at the end, but that should be pretty efficient. Everything that Galaxy is going to see in the end is the re-assembled file path, everything else is either handled by a WSGI middleware (not very efficient and going to block connections, but a good out of the box solution), an nginx lua module (https://github.com/mmatuska/lua-tus-server) or the official tusd server (or any of the other implementations).

This is the protocol used at vimeo for instance, I'd assume this is going to scale pretty well at a much larger scale (apples and oranges of course, since they're probably directly writing to cloud storage).

[hexylena]

Ah that's very encouraging to hear! tusd sounds promising