image share bulk uploading is slow and unreliable

[joernhees]

Steps to reproduce

1. setup small debian 9 vps with snap install nextcloud
2. create some new folder to share
3. click share, share link enable, allow uploading and editing, password protect
4. send link & pw to family/friends
5. have them upload their photos & videos of your vacation (often > 1000 files, usually < 10 GB)

Expected behaviour

It just works (in Firefox & Chrome), i don't get emails from my mom about it not working / aborting all the time and her giving up.

That means:

1. Reliable upload with client (JS) side fault tolerance to circumvent possibly upcoming problems (e.g., potential server timeouts (think of a large gopro video)).
2. Parallelized uploads (e.g.) to work around sequential file overheads that seem significant even for ~300 KB files (see 10 tab workaround mentioned below).

All in all, i don't expect wonders, but ~5 GB in many small and some big files should be uploadable with a 6 Mbit upstream in about 3-4 hours (1.8 h in theory).

Actual behaviour

Usually i get tons of emails complaining that upload is very slow and after they left it on overnight it "finished", then me telling them "are you sure you only wanted to upload till day 2 of 7?" - "what???" - "aah, sorry, seems it aborted somewhere, please keep trying till it's all there for the next 365 days"...

Caricature aside, what i can observe happens very often: very slow upload speed ~ 100 KB/s (~ 700 KB/s expected), sequential upload, aborted somewhere without clear indication for user.

I have actually worked around this myself by opening 10 tabs of the upload folder and then dragging a chunk into each... voila, 700 KB/s upload speed. That's however nothing i can expect of not so IT savvy people :-/.

I have observed this over many different webspace hosters in the past. I even just set up a separate VM with snap exactly to rule out "misconfiguration". I'll happily test your server / try things out.

Server configuration

Operating system:
debian 9

Web server:
apache (snap)

Database:
mysql 5.7.22 (snap)

PHP version:
7.1.26 (snap)

Nextcloud version: (see Nextcloud admin page)
14.0.5

Updated from an older Nextcloud/ownCloud or fresh install:
fresh

Where did you install Nextcloud from:
snap

Signing status

No errors have been found.

App list

Enabled:
  - accessibility: 1.0.1
  - activity: 2.7.0
  - calendar: 1.6.4
  - cloud_federation_api: 0.0.1
  - comments: 1.4.0
  - contacts: 2.1.8
  - dav: 1.6.1
  - federatedfilesharing: 1.4.0
  - federation: 1.4.0
  - files: 1.9.0
  - files_pdfviewer: 1.3.2
  - files_sharing: 1.6.2
  - files_texteditor: 2.6.0
  - files_trashbin: 1.4.1
  - files_versions: 1.7.1
  - files_videoplayer: 1.3.0
  - firstrunwizard: 2.3.0
  - gallery: 18.1.0
  - logreader: 2.0.0
  - lookup_server_connector: 1.2.0
  - nextcloud_announcements: 1.3.0
  - notifications: 2.2.1
  - oauth2: 1.2.1
  - password_policy: 1.4.0
  - provisioning_api: 1.4.0
  - serverinfo: 1.4.0
  - sharebymail: 1.4.0
  - support: 1.0.0
  - survey_client: 1.2.0
  - systemtags: 1.4.0
  - theming: 1.5.0
  - twofactor_backupcodes: 1.3.1
  - workflowengine: 1.4.0
Disabled:
  - admin_audit
  - encryption
  - files_external
  - user_external
  - user_ldap

Config report

{
    "system": {
        "apps_paths": [
            {
                "path": "\/snap\/nextcloud\/current\/htdocs\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/snap\/nextcloud\/current\/nextcloud\/extra-apps",
                "url": "\/extra-apps",
                "writable": true
            }
        ],
        "supportedDatabases": [
            "mysql"
        ],
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "memcache.local": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0
        },
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost",
            "cloud.joernhees.de"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "14.0.5.2",
        "overwrite.cli.url": "http:\/\/localhost",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***"
    },
    "apps": {
        "accessibility": {
            "enabled": "yes",
            "installed_version": "1.0.1",
            "types": ""
        },
        "activity": {
            "enabled": "yes",
            "installed_version": "2.7.0",
            "types": "filesystem"
        },
        "backgroundjob": {
            "lastjob": "224"
        },
        "calendar": {
            "enabled": "yes",
            "installed_version": "1.6.4",
            "types": ""
        },
        "cloud_federation_api": {
            "enabled": "yes",
            "installed_version": "0.0.1",
            "types": "filesystem"
        },
        "comments": {
            "enabled": "yes",
            "installed_version": "1.4.0",
            "types": "logging"
        },
        "contacts": {
            "enabled": "yes",
            "installed_version": "2.1.8",
            "types": ""
        },
        "core": {
            "backgroundjobs_mode": "cron",
            "installedat": "1547923568.4601",
            "lastcron": "1547991765",
            "lastupdatedat": "1547923568.4644",
            "public_files": "files_sharing\/public.php",
            "public_webdav": "dav\/appinfo\/v1\/publicwebdav.php",
            "scss.variables": "7a173342cb0e112d3b11053e87e64a8e",
            "vendor": "nextcloud"
        },
        "dav": {
            "enabled": "yes",
            "installed_version": "1.6.1",
            "types": "filesystem"
        },
        "federatedfilesharing": {
            "enabled": "yes",
            "installed_version": "1.4.0",
            "types": ""
        },
        "federation": {
            "enabled": "yes",
            "installed_version": "1.4.0",
            "types": "authentication"
        },
        "files": {
            "cronjob_scan_files": "500",
            "enabled": "yes",
            "installed_version": "1.9.0",
            "types": "filesystem"
        },
        "files_pdfviewer": {
            "enabled": "yes",
            "installed_version": "1.3.2",
            "types": ""
        },
        "files_sharing": {
            "enabled": "yes",
            "installed_version": "1.6.2",
            "types": "filesystem"
        },
        "files_texteditor": {
            "enabled": "yes",
            "installed_version": "2.6.0",
            "types": ""
        },
        "files_trashbin": {
            "enabled": "yes",
            "installed_version": "1.4.1",
            "types": "filesystem,dav"
        },
        "files_versions": {
            "enabled": "yes",
            "installed_version": "1.7.1",
            "types": "filesystem,dav"
        },
        "files_videoplayer": {
            "enabled": "yes",
            "installed_version": "1.3.0",
            "types": ""
        },
        "firstrunwizard": {
            "enabled": "yes",
            "installed_version": "2.3.0",
            "types": "logging"
        },
        "gallery": {
            "enabled": "yes",
            "installed_version": "18.1.0",
            "types": ""
        },
        "logreader": {
            "enabled": "yes",
            "installed_version": "2.0.0",
            "ocsid": "170871",
            "types": ""
        },
        "lookup_server_connector": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "authentication"
        },
        "nextcloud_announcements": {
            "enabled": "yes",
            "installed_version": "1.3.0",
            "pub_date": "Sat, 10 Dec 2016 00:00:00 +0100",
            "types": "logging"
        },
        "notifications": {
            "enabled": "yes",
            "installed_version": "2.2.1",
            "types": "logging"
        },
        "oauth2": {
            "enabled": "yes",
            "installed_version": "1.2.1",
            "types": "authentication"
        },
        "password_policy": {
            "enabled": "yes",
            "installed_version": "1.4.0",
            "types": ""
        },
        "provisioning_api": {
            "enabled": "yes",
            "installed_version": "1.4.0",
            "types": "prevent_group_restriction"
        },
        "serverinfo": {
            "enabled": "yes",
            "installed_version": "1.4.0",
            "types": ""
        },
        "sharebymail": {
            "enabled": "yes",
            "installed_version": "1.4.0",
            "types": "filesystem"
        },
        "support": {
            "enabled": "yes",
            "installed_version": "1.0.0",
            "types": ""
        },
        "survey_client": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": ""
        },
        "systemtags": {
            "enabled": "yes",
            "installed_version": "1.4.0",
            "types": "logging"
        },
        "theming": {
            "enabled": "yes",
            "installed_version": "1.5.0",
            "types": "logging"
        },
        "twofactor_backupcodes": {
            "enabled": "yes",
            "installed_version": "1.3.1",
            "types": ""
        },
        "workflowengine": {
            "enabled": "yes",
            "installed_version": "1.4.0",
            "types": "filesystem"
        }
    }
}

Are you using external storage: no

Are you using encryption: no

Are you using an external user-backend: no

Client configuration

Browser:
Chrome / Firefox

Operating system:
Win / Mac / Linux

Logs

Web server error log

Web server error log

hard to remove sensitive information from, will provide on demand

Nextcloud log (data/nextcloud.log)

Nextcloud log

nothing alarming

Browser log

Browser log

will provide on demand

[joernhees]

some further digging: browser network log show sequential PUTs on chrome and firefox with a 401 GET request to /index.php/apps/files/ajax/getstoragestats.php in between each PUT

[kesselb]

server/apps/files/js/files.js

Line 35 in 576e44f

updateStorageQuotas: function() {

Yay in this function a check of OC.currentUser is missing. I guess this could be done like updateStorageStatistics in the same file (move logic to _updateStorageQuotas and add a wrapper).

Pull Requests are always welcome 😃

[joernhees]

don't really know what i'm doing, but yeah, i think that part is actually only a side to the main problem here, let's discuss that further in #13697

[MorrisJobke]

Part of the slowness is resolved with #13855

[tomasz-grobelny]

The reliability part might (at least in part) be connected to the fact that UI throws a lot of MKCOL request at server at the same time. This may lead to timeouts (502 from server) which are not handled in any sane way. @joernhees, could you tell a little more about structure of the folder you are trying to upload? Is it just one folder with multiple files or are there also multiple subfolders inside the main folder?

[evilham]

The reliability part could also be because of #10527 solved in #14674.

LIkely this use-case involves a bunch of Thumbs.db or DCM_0001.jpg or similar (files with the same name in different directories).

[kesselb]

@joernhees Have you had a chance to test Nextcloud 16? I hope we made some progress here 🚀

[ghost]

This issue has been automatically marked as stale because it has not had recent activity and it seems to be missing some essential informations. It will be closed if no further activity occurs. Thank you for your contributions.

[joernhees]

The reliability part could also be because of #10527 solved in #14674.

LIkely this use-case involves a bunch of Thumbs.db or DCM_0001.jpg or similar (files with the same name in different directories).

this was just a single folder, no dup filenames...

@joernhees Have you had a chance to test Nextcloud 16? I hope we made some progress here 🚀

@kesselb sorry lacking the kind of time atm to switch my (snap) stable (i think 14) nextcloud to 16 (and deal with problems should they pop up)... i tried uploading random image datasets to your demo instances (16.0.3) though (on a 1 Gbit/s connection):

dataset	results
http://vis-www.cs.umass.edu/lfw/lfw.tgz (5749 folders, few jpegs each, < 20 KB each)	371 KB/s (tons of errors)
https://thementalclub.com/3d-wallpapers-in-a-zip-file-7396 (picked the nature one, ~8 GB of 10 - 15 MB per pic)	40 MB/s (but demo inst out of space after a few seconds)

with the 2nd dataset i noticed that it keeps sending images over to the server even after the "your storage is full" error... maybe it could abort the current batch in such cases instead of actually sending all remaining images...

anyhow, i think the performance problem (if not also errors) already shows with the first dataset. Maybe someone already running nextcloud 16 could briefly check the second?

[evilham]

This shouldn't be closed.

[szaimen]

Hi, please update to 24.0.8 or better 25.0.2 and report back if it fixes the issue. Thank you!

[evilham]

Hah, thanks for pinging, it's certainly been a while. Will schedule testing this over the next few days.

[szaimen]

Thank you for checking! :)

[evilham]

TL;DR: The issue persists on 25.0.2, uploading small files does the whole HTTP connection dance and loads the CPU in a noticeable fashion on the server, all while introducing disproportionate delays on such file sets which reduces expected performance from ~50MBps to ~ 180KBps (aka to 0.4% of the behaviour with big files).

Hey, just did a quick test with the first dataset (the problematic one) and observed the same behaviour on version 25.0.2 as ~4 years ago.

The rough statistics are:

• dataset size: 179.6Mb
• achieved data transfer speed as reported by NC's UI:180KBps
• rough total upload time: 23 Mins

The larger dataset is no longer available apparently, but just as comparison: this is a server on the LAN which is able to achieve significantly higher upload speeds on big files of very stable 50 MBps as reported by NC's UI.

The UX being: uploading two 2Gb files takes significantly less than the one folder of 180Mb.