[Bug]: NC 28 Cannot upload file larger than 2GB (javascript error) #42704
Comments
Trevelian
added
0. Needs triage
|
Cc @nextcloud/server-frontend |
|
We need to upload using streams / file objects rather than loading the file into a buffer. Especially for low memory devices. |
|
Hmmm, I did some tests a while back to ensure this doesn't happen. https://github.com/nextcloud-libraries/nextcloud-upload/blob/ec7c2bd1df51fee8c40f9f413088b69be55a4592/lib/uploader.ts#L207 @Trevelian can you tell us if you disabled chunk uploading? |
|
I am having the same problem - when chunk uploading is disabled, the whole file is loaded into RAM until the browser cuts it off at the Javascript memory limit. Seeing huge RAM usage when uploading large files with chunking back on (200MB chunks). |
|
Well well well, yeah that make sense! |
|
I'm setting up an NC 27 to test this, but I think it was working before. We've had chunking off in order to upload large (>30GB) files without risking timeouts on the reassembling for years. |
|
NC 27, chunks off, 4GiB uploaded. No extraordinary memory usage. |
|
@skjnldsv Yes my NC server before 28 was working far more better with max_chunk_size at "0" |
|
ALright, thanks everyone for the help, setting this as |
skjnldsv
added
1. to develop
|
Great! Thanks. I'm back on 27 for now, but happy to test out any updates you might have. |
|
Just tested 27 and RAM is increased by a lot (looks like twice the uploaded file size) Peek.23-01-2024.17-43.mp4I do wonder if we shouldn't have a cleaner limit and communicate it to the front end if chunk upload is disabled. |
@AdamVenn and you never had RAM issues? Tested on older NC version (see above), it would still load the file in the browser memory, so 30GB is crazy to me 🙈 |
The fix for the main post is here: #42704 Regarding the RAM usage, it seems it was there all along anyway, so I'd say we'll have to investigate a bit further and see what we can do |
|
@skjnldsv Its strange, when I test uploading a big file on NC27 with max_chunk_size to 0 in my case I dont have any ram usage like NC28 and the upload succed.
|
|
@Trevelian Firefox? |
|
NC 28.0.1, bug is still present. Our biggest upload in a single file is 23 GB. So the 30 GB mentioned above are not so unusual. I did some more testing and were able to successfully upload a 20 GB file with a chunk size of 25 GB (several smaller values work too; a reload of php-fpm might be necessary while doing these test). So I guess as long as the chunk size does not equal zero, the uploads work. However, the file still has to be moved from the temporary directory to the data directory, which delays the operation in comaprison to a chunk size of zero. |
|
@LM-vb a chunk size of 25GB makes zero sense. Anyway, this is fixed on 28.0.2 |
|
I tested with version 28.0.2 and its not fixed. (firefox) Switched back chunck to default value of 10MB. (which is probably better) |
|
28.0.2.5 still has the bug. Is this still being worked on? Where can we see the progress? |
|
will be fixed in 28.0.3 |
Bug description
When trying to upload a file larger than 2GB the transfer never start (no bandwidth used) and we go this error on the web browser console :
Uncaught TypeError: Blob constructor: ArrayBuffer branch of ((ArrayBufferView or ArrayBuffer) or Blob or USVString) can't be an ArrayBuffer or an ArrayBufferView larger than 2 GB
index-f812dc31.mjs:1485:19
Steps to reproduce
Expected behavior
Successfully uploading big files
Installation method
Community Manual installation with Archive
Nextcloud Server version
28
Operating system
RHEL/CentOS
PHP engine version
PHP 8.2
Web server
Apache (supported)
Database engine version
MariaDB
Is this bug present after an update or on a fresh install?
Upgraded to a MAJOR version (ex. 22 to 23)
Are you using the Nextcloud Server Encryption module?
None
What user-backends are you using?
Configuration report
{ "system": { "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "nexcloud.server.be" ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "dbtype": "mysql", "allow_user_to_change_display_name": false, "forcessl": true, "version": "28.0.1.1", "default_language": "fr", "force_language": "fr", "default_locale": "fr_BE", "force_locale": "fr_BE", "dbname": "***REMOVED SENSITIVE VALUE***", "upgrade.disable-web": true, "dbhost": "***REMOVED SENSITIVE VALUE***", "proxy": "proxy.server.be:9999", "overwritewebroot": "\/", "overwriteprotocol": "https", "overwritehost": "nexcloud.server.be", "overwrite.cli.url": "https:\/\/nexcloud.server.be\/", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "profile.enabled": false, "trashbin_retention_obligation": "auto, 30", "updatechecker": false, "appstoreenabled": true, "dbtableprefix": "oc_", "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "logtimezone": "Europe\/Brussels", "logfile": "\/var\/log\/nextcloud\/nextcloud.log", "default_phone_region": "BE", "installed": true, "skeletondirectory": "", "log_authfailip": true, "forceSSLforSubdomains": true, "filesystem_check_changes": 0, "ldapIgnoreNamingRules": false, "ldapUserCleanupInterval": 51, "theme": "", "loglevel": 2, "maintenance": false, "htaccess.RewriteBase": "\/", "ldapProviderFactory": "\\OCA\\User_LDAP\\LDAPProviderFactory", "memcache.local": "\\OC\\Memcache\\APCu", "memcache.locking": "\\OC\\Memcache\\Redis", "redis": { "host": "***REMOVED SENSITIVE VALUE***", "port": 0, "dbindex": 0, "timeout": 1.5 }, "data-fingerprint": "33d0bd49f33c240b0dbc223f0c283dfb", "mail_domain": "***REMOVED SENSITIVE VALUE***", "mail_smtpmode": "smtp", "objectstore": { "class": "\\OC\\Files\\ObjectStore\\S3", "arguments": { "bucket": "nextcloud", "key": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "hostname": "s3.server.be", "port": 443, "use_ssl": true, "region": "us-east-1", "legacy_auth": false, "autocreate": false, "use_path_style": false } }, "app_install_overwrite": [ "passman", "files_versions_s3", "user_saml" ], "mysql.utf8mb4": true, "simpleSignUpLink.shown": false } }List of activated Apps
Nextcloud Signing status
Nextcloud Logs
No response
Additional info
No response
The text was updated successfully, but these errors were encountered: