Slow uploads with HTTP/2 #9578

haarp · 2021-01-23T23:26:47Z

How to use GitHub

Please use the 👍 reaction to show that you are affected by the same issue.
Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
Subscribe to receive notifications on status change and new comments.

Summary

When HTTP/2 is enabled on the webserver, uploads are very slow, topping out at slightly over 1MByte/s, despite both client and server offering vastly more bandwidth. Once HTTP2 was disabled, upload speeds were improved by a factor of over x10.

All PHP/DB optimizations are in place, and more. The CPU load on server and client remains low, IO load on the DB and storage remains low. Issue is observed during large file uploads, so DB overhead isn't the issue.

This issue was observed by multiple people, e.g. here, here and here. It seems to affect both nginx and Apache.

Steps to reproduce

Enable HTTP/2 on webserver
Upload using a WebDAV client - this test used the Windows 10 client via a mapped network drive
Get uploadin'

Expected behaviour

>10 Mbyte/s

Actual behaviour

1-2 Mbyte/s

Server configuration

Operating system: Debian Linux 10
Web server: Apache 2.4.38-3+deb10u4
Database: MariaDB 10.3.27-0+deb10u1
PHP version: PHP FPM-CGI 7.3.19-1~deb10u1
Nextcloud version: 19.0.7
Updated from an older Nextcloud/ownCloud or fresh install: Updated
Where did you install Nextcloud from: Included updater

Signing status:

Signing status

No errors have been found.

List of activated apps:

App list

Enabled:
  - accessibility: 1.5.0
  - activity: 2.12.1
  - calendar: 2.0.4
  - camerarawpreviews: 0.7.10
  - checksum: 0.4.5
  - cloud_federation_api: 1.2.0
  - comments: 1.9.0
  - contacts: 3.4.3
  - dav: 1.15.0
  - duplicatefinder: 0.0.4
  - federatedfilesharing: 1.9.0
  - federation: 1.9.0
  - files: 1.14.0
  - files_3d: 0.3.3
  - files_rightclick: 0.16.0
  - files_sharing: 1.11.0
  - files_trashbin: 1.9.0
  - files_versions: 1.12.0
  - logreader: 2.4.0
  - lookup_server_connector: 1.7.0
  - metadata: 0.12.0
  - nextcloud_announcements: 1.8.0
  - notes: 3.6.4
  - notifications: 2.7.0
  - oauth2: 1.7.0
  - password_policy: 1.9.1
  - photos: 1.1.0
  - privacy: 1.3.0
  - provisioning_api: 1.9.0
  - recommendations: 0.7.0
  - serverinfo: 1.9.0
  - settings: 1.1.0
  - sharebymail: 1.9.0
  - support: 1.2.1
  - survey_client: 1.7.0
  - systemtags: 1.9.0
  - text: 3.0.1
  - theming: 1.10.0
  - twofactor_backupcodes: 1.8.0
  - updatenotification: 1.9.0
  - viewer: 1.3.0
  - workflowengine: 2.1.0
Disabled:
  - admin_audit
  - contactsinteraction
  - encryption
  - files_external
  - files_pdfviewer
  - files_videoplayer
  - firstrunwizard
  - ransomware_protection
  - user_ldap

Nextcloud configuration:

Config report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
        "overwritewebroot": "\/",
        "htaccess.RewriteBase": "\/",
        "dbtype": "mysql",
        "version": "19.0.7.1",
        "knowledgebaseenabled": false,
        "apps_paths": [
            {
                "path": "\/var\/www\/nextcloud\/apps",
                "url": "\/apps",
                "writable": true
            }
        ],
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "loglevel": 2,
        "mail_smtpmode": "sendmail",
        "theme": "",
        "maintenance": false,
        "trashbin_retention_obligation": "auto",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "session_lifetime": 86400,
        "mysql.utf8mb4": true,
        "updater.release.channel": "stable",
        "preview_max_x": "2048",
        "preview_max_y": "2048",
        "preview_max_filesize_image": 100,
        "app_install_overwrite": [
            "checksum"
        ]
    }
}

Are you using external storage, if yes which one: no
Are you using encryption: no
Are you using an external user-backend, if yes which one: no

Client configuration

Browser: -
Client software: Windows 10 WebClient service (Microsoft-WebDAV-MiniRedir/10.0.18363)
Operating system: Windows 10

Logs

Web server error log

no error logs during upload

Nextcloud log (data/nextcloud.log)

Nextcloud log

no nextcloud log during upload

Browser log

not using a browser

The text was updated successfully, but these errors were encountered:

aledr · 2021-06-02T02:09:38Z

I see no improvement in disabling h2 and h2c.
1.8 MB/sec maximum.

Apache 2.4.43
PHP 7.4.19
Nextcloud 21.0.2

solracsf · 2022-01-02T15:08:38Z

This seems to be an overall issue, not a Nextcloud (only) issue.

See here for more technical informations
https://blog.cloudflare.com/delivering-http-2-upload-speed-improvements/

ghost · 2022-03-11T17:57:15Z

I could confirm that it is not a Nextcloud issue, but rather an issue with webservers ( nginx, apache ).

Using the following configuration solved the upload bandwidth issue with nginx

client_body_buffer_size 512k;

CarlSchwan · 2022-05-18T08:27:22Z

#8323

zijiren233 · 2022-10-17T15:09:09Z

For apache, add the H2WindowSize 1048576 parameter
For nginx,add the http2_body_preread_size 1048576; parameter
Both of these can be set as global parameters

sentient-russ · 2022-10-23T15:29:19Z

TY zijiren233! This allowed the same improvement while keeping https end to end encryption. I did the following:

#If your using SSLEngine on in your VirtualHost configuration for nextcloud make sure mod http2 is enabled

sudo a2enmod http2

sudo nano /etc/apache2/mods-enabled/http2.conf

#Add line H2WIndowSize 1048576

<IfModule !mpm_prefork>
    Protocols h2 h2c http/1.1
    H2WindowSize 1048576
</IfModule>

Jookia · 2022-10-30T21:37:47Z

Just to add the pile of data, we went from 300KiB/s to 6MiB/s by setting client_max_body_size to 5M; A smaller buffer size did not max out the connection.

prashant3285 · 2022-11-06T11:37:56Z

TY zijiren233! This allowed the same improvement while keeping https end to end encryption. I did the following:

#If your using SSLEngine on in your VirtualHost configuration for nextcloud make sure mod http2 is enabled

sudo a2enmod http2

sudo nano /etc/apache2/mods-enabled/http2.conf

#Add line H2WIndowSize 1048576
<IfModule !mpm_prefork>
    Protocols h2 h2c http/1.1
    H2WindowSize 1048576
</IfModule>

Thanks @zijiren233 and @sentient-russ for the guidance.
I tried H2WindowSize 1048576 but didnt see much improvement.
My connection upload limit is 3.5 MB/s but I was getting 1 MB/s
So I took trials and finally settled at H2WindowSize 5242880 . Now I am getting 3.5 MB/s as next cloud upload speed.

Also I disabled file chunking as it was causing severe delays in uploading Check Here
sudo -u www-data php var/www/nextcloud/occ config:app:set files max_chunk_size --value 0

Edit:
uploading big files was becoming an issue so moved to chunk size of 50Mb
sudo -u www-data php var/www/nextcloud/occ config:app:set files max_chunk_size --value 5242880

rs-development · 2022-11-12T14:39:19Z

So does anyone experience issues with downloads or only uploads?

I am using haproxy in front of nginx/fpm. If I have enabled http2 for all connections downloads start of 50% and drop down to 300kB/s. With http2 terminated in haproxy and http/1.1 between haproxy and nginx the download maxes out my 50MBit/s connection.

szaimen · 2022-11-12T15:12:02Z

H2WindowSize

Could someone please open a PR over at https://github.com/nextcloud/documentation to document that this value needs to be adjusted when http2 is activated in Apache? Thanks!

szaimen · 2023-01-23T10:41:46Z

I'll move this to documentation so that it does not get lost

reference: - nextcloud/documentation#9578 - https://blog.cloudflare.com/delivering-http-2-upload-speed-improvements/ BEFORE: Web and WebDav upload speed 18-20MB/s ··· client_max_body_size 0; client_body_buffer_size 0; ··· AFTER: Web and WebDav upload speed 60-80MB/s ··· client_max_body_size 0; client_body_buffer_size 512k; http2_body_preread_size 1048576; ···

haarp added 0. Needs triage bug labels Jan 23, 2021

This comment has been minimized.

Sign in to view

szaimen added the needs info label Jul 7, 2021

This comment has been minimized.

Sign in to view

szaimen added 1. to develop 0. Needs triage needs info and removed 0. Needs triage needs info 1. to develop labels Jul 27, 2021

This comment has been minimized.

Sign in to view

zamentur mentioned this issue Dec 28, 2021

Be able to deploy /e/ environement YunoHost/issues#1954

Open

4 tasks

This comment has been minimized.

Sign in to view

solracsf mentioned this issue Jan 2, 2022

Slow file upload with HTTP/2 nextcloud/server#30464

Closed

CarlSchwan mentioned this issue May 18, 2022

Improve Upload speed with nginx #8323

Merged

solracsf added the stale label Sep 29, 2022

szaimen transferred this issue from nextcloud/server Jan 23, 2023

szaimen mentioned this issue Jan 23, 2023

[Bug]: Upload using web interface has continuous long pauses without doing anything nextcloud/server#34781

Closed

9 tasks

joshtrichards added manual: admin and removed stale labels Aug 26, 2023

aquaspy mentioned this issue Sep 9, 2023

Fix slow uploads under HTTP2 YunoHost-Apps/nextcloud_ynh#602

Closed

joshtrichards added feature: install and update performance 🚀 labels Sep 15, 2023

Acconut mentioned this issue Apr 19, 2024

Given nginx configuration slows down uploads over https tus/tusd#1106

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Slow uploads with HTTP/2 #9578

Slow uploads with HTTP/2 #9578

haarp commented Jan 23, 2021 •

edited

Loading

aledr commented Jun 2, 2021

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

solracsf commented Jan 2, 2022 •

edited

Loading

ghost commented Mar 11, 2022 •

edited by ghost

Loading

CarlSchwan commented May 18, 2022

zijiren233 commented Oct 17, 2022 •

edited

Loading

sentient-russ commented Oct 23, 2022

Jookia commented Oct 30, 2022

prashant3285 commented Nov 6, 2022 •

edited

Loading

rs-development commented Nov 12, 2022

szaimen commented Nov 12, 2022

szaimen commented Jan 23, 2023

Slow uploads with HTTP/2 #9578

Slow uploads with HTTP/2 #9578

Comments

haarp commented Jan 23, 2021 • edited Loading

How to use GitHub

Summary

Steps to reproduce

Expected behaviour

Actual behaviour

Server configuration

Client configuration

Logs

Web server error log

Nextcloud log (data/nextcloud.log)

Browser log

aledr commented Jun 2, 2021

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

solracsf commented Jan 2, 2022 • edited Loading

ghost commented Mar 11, 2022 • edited by ghost Loading

CarlSchwan commented May 18, 2022

zijiren233 commented Oct 17, 2022 • edited Loading

sentient-russ commented Oct 23, 2022

Jookia commented Oct 30, 2022

prashant3285 commented Nov 6, 2022 • edited Loading

rs-development commented Nov 12, 2022

szaimen commented Nov 12, 2022

szaimen commented Jan 23, 2023

haarp commented Jan 23, 2021 •

edited

Loading

solracsf commented Jan 2, 2022 •

edited

Loading

ghost commented Mar 11, 2022 •

edited by ghost

Loading

zijiren233 commented Oct 17, 2022 •

edited

Loading

prashant3285 commented Nov 6, 2022 •

edited

Loading