Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2.6.0: issues with HTTP/2 #7610

Closed
fritzmg opened this issue Nov 27, 2019 · 15 comments
Closed

2.6.0: issues with HTTP/2 #7610

fritzmg opened this issue Nov 27, 2019 · 15 comments

Comments

@fritzmg
Copy link

@fritzmg fritzmg commented Nov 27, 2019

In 2.5.4, HTTP/2 was disabled in the ownCloud client due to issues within the Qt library. In 2.6.0 HTTP/2 was re-enabled again, as the issues were supposedly fixed.

However, when using the ownCloud client in version 2.6.0 now, syncing does not work with our ownCloud instance anymore, when there is a large number of files to be checked/synced. When using a PHP-FPM setup, we get the following error message within the client:

Server stopped accepting new streams before this stream was established

This error weirdly does not appear, when logging is enabled in the client. The client only times out after the same amount of time when that error would happen without logging. The log entries when logging is enabled are:

11-26 12:21:35:984 [ warning sync.networkjob ]:	Network job timeout QUrl("https://owncloud.example.org/remote.php/dav/files/ownclouduser/some/folder")
11-26 12:21:35:984 [ warning sync.networkjob ]:	QNetworkReply::OperationCanceledError "Connection timed out" QVariant(Invalid)
11-26 12:21:35:984 [ info sync.networkjob.lscol ]:	LSCOL of QUrl("https://owncloud.example.org/remote.php/dav/files/ownclouduser/some/folder") FINISHED WITH STATUS "OperationCanceledError Connection timed out"
11-26 12:21:35:984 [ warning sync.discovery ]:	LSCOL job error "Operation canceled" 0 QNetworkReply::OperationCanceledError
11-26 12:21:35:984 [ warning sync.discovery ]:	Server error in directory "some/folder" 0
11-26 12:21:35:984 [ info sync.engine ]:	Sync run took  328229 ms
11-26 12:21:35:984 [ info gui.socketapi ]:	Sending SocketAPI message --> "STATUS:OK:…\\ownCloud" to QLocalSocket(0x796efb0)
11-26 12:21:35:984 [ debug sync.localdiscoverytracker ]	[ OCC::LocalDiscoveryTracker::slotSyncFinished ]:	sync failed, keeping last sync's local discovery path list
11-26 12:21:35:984 [ debug sync.networkjob ]	[ OCC::AbstractNetworkJob::slotFinished ]:	Network job OCC::LsColJob finished for "/some/folder"
11-26 12:21:35:984 [ info gui.folder ]:	Client version 2.6.0 (build 12644)  Qt 5.12.5  SSL  OpenSSL 1.1.1d  10 Sep 2019
11-26 12:21:35:984 [ warning gui.folder ]:	SyncEngine finished with ERROR
11-26 12:21:35:984 [ info gui.folder ]:	Folder sync result:  2
11-26 12:21:35:984 [ info gui.folder ]:	the last 1 syncs failed
11-26 12:21:35:984 [ info gui.socketapi ]:	Sending SocketAPI message --> "STATUS:OK:…\\ownCloud" to QLocalSocket(0x796efb0)
11-26 12:21:35:984 [ info gui.socketapi ]:	Sending SocketAPI message --> "UPDATE_VIEW:…\\ownCloud" to QLocalSocket(0x796f0e0)
11-26 12:21:35:984 [ info gui.socketapi ]:	Sending SocketAPI message --> "UPDATE_VIEW:…\\ownCloud" to QLocalSocket(0x796efb0)
11-26 12:21:35:984 [ info gui.application ]:	Sync state changed for folder  "https://owncloud.example.org/remote.php/dav/files/ownclouduser/" :  "Error"

On the server side, this is the only error that coincides:

[Tue Nov 26 17:01:19.811110 2019] [proxy_fcgi:error] [pid 24501] (32)Broken pipe: [client 212.186.187.58:48992] AH01075: Error dispatching request to : (passing brigade to output filters)

There are no PHP, PHP-FPM or MySQL errors present in the respective error logs otherwise.

Since the source error message ("Server stopped accepting new streams before this stream was established") comes from within the Qt library here, it looks like it has something to do with HTTP/2 again.

And sure enough, after disabling HTTP/2 on the server side, no synchronisation errors happened any more.

Server configuration

Operating system:
Ubuntu 16.04.6

Web server:
Apache 2.4.18
nginx 1.16.1.3

Database:
MySQL 5.7.28

PHP version:
7.3.12

ownCloud version:
10.3.1

Client configuration

Client version:
2.6.0

Operating system:
Windows

OS language:
English

Installation path of client:
C:\Program Files (x86)\ownCloud

@guruz guruz added the sev2-high label Nov 27, 2019
@guruz guruz assigned guruz, ckamm and ogoffart and unassigned guruz Nov 27, 2019
@guruz guruz added this to the 2.6.1 milestone Nov 27, 2019
@guruz

This comment has been minimized.

Copy link
Collaborator

@guruz guruz commented Nov 29, 2019

@fritzmg Thank you for the report. This sounds unfortunate.

@ckamm Do you think this could be https://bugreports.qt.io/browse/QTBUG-77852 ?

@guruz

This comment has been minimized.

Copy link
Collaborator

@guruz guruz commented Nov 29, 2019

@fritzmg Just wondering which Apache version exactly you use?
And which model in Apache (worker, MPM, ...)
I'm also curious about the FPM version

@felixboehm

This comment has been minimized.

Copy link
Contributor

@felixboehm felixboehm commented Nov 29, 2019

ownClouds running always with mod prefork, which doesn‘t support http/2.
It would be interesting to have a test with php fpm using http/2, also in terms of speed ...

@fritzmg

This comment has been minimized.

Copy link
Author

@fritzmg fritzmg commented Dec 1, 2019

@fritzmg Just wondering which Apache version exactly you use?

2.4.18

However, the issue is also present when using nginx only (version 1.16.1)

And which model in Apache (worker, MPM, ...)

Hm, not sure, how do I check that?

I'm also curious about the FPM version

Not sure what you mean there - isn't that just the PHP version?

@michaelstingl

This comment has been minimized.

Copy link
Collaborator

@michaelstingl michaelstingl commented Dec 2, 2019

@TheOneRing please disable again for the 2.6.1

@fritzmg

This comment has been minimized.

Copy link
Author

@fritzmg fritzmg commented Dec 2, 2019

Before you revert anything: can anybody else confirm the issue? This might still be a server configuration issue. As far as I know, I have done every conceivable and recommended server configuration - but there might still be something, that needs to be done. I just need to find out what it is.

@jnweiger

This comment has been minimized.

Copy link
Contributor

@jnweiger jnweiger commented Dec 2, 2019

I haven't tried HTTP2 myself, but we collected some info. It probably requires php-FPM in any case, which is not our recommended setup. There are two use cases

  • the ownCloud server itself does the FPM so that HTTP2 can work,
  • a reverse proxy (e.g. nginx) does HTTP2 -- in this case the issue can be considered independant of support in the ownCloud server.
@fritzmg

This comment has been minimized.

Copy link
Author

@fritzmg fritzmg commented Dec 2, 2019

It probably requires php-FPM in any case, which is not our recommended setup.

Why is that? The official documentation does not mention that anywhere, at least as far as I see. PHP-FPM should always be the recommended way of running PHP.

@michaelstingl

This comment has been minimized.

Copy link
Collaborator

@michaelstingl michaelstingl commented Dec 2, 2019

@fritzmg here you can find some background: owncloud/core#30725

@fritzmg

This comment has been minimized.

Copy link
Author

@fritzmg fritzmg commented Dec 2, 2019

@michaelstingl interesting, thanks. But still, the issues I am having also happen with a pure nginx + PHP-FPM setup, while owncloud/core#30725 mostly talks about Apache support of HTTP/2.

@ho4ho

This comment has been minimized.

Copy link

@ho4ho ho4ho commented Dec 2, 2019

@HanaGemela HanaGemela added the bug label Dec 3, 2019
TheOneRing added a commit that referenced this issue Dec 3, 2019
Issue: #7610
@michaelstingl

This comment has been minimized.

Copy link
Collaborator

@michaelstingl michaelstingl commented Dec 3, 2019

@fritzmg for the upcoming 2.6.1 release, we removed HTTP/2 support, but you can enable it for testing using the environment variable OWNCLOUD_HTTP2_ENABLED . No ETA a yet to bring it back…

@guruz

This comment has been minimized.

Copy link
Collaborator

@guruz guruz commented Dec 3, 2019

@tessus Have you seen this in your testing?

@tessus

This comment has been minimized.

Copy link
Contributor

@tessus tessus commented Dec 3, 2019

@guruz no, but I didn't do extensive testing out of fear. IMO a lot of VFS issues bled over to the basic sync code for some reason and the entire client is less stable than before.

Currently I'm still using an ancient version for 2 reasons: h2 works, no VFS code:

Version 2.4.3 (build 10347).
Built from Git revision 560555 on Aug 10 2018, 21:27:04 using Qt 5.10.1, OpenSSL 1.0.2n 7 Dec 2017

But let me comment on a few things here, because I have tthe feeling that some people do not understand h2, backend, and fpm.

  • h2 REQUIRES a worker or event MPM
  • worker or event REQUIRES php-fpm

h2 uses a thread model, which will break when using a prefork MPM on the web server, which is process based. Furthermore a process based server has several other disadvantages as well: one php version per server, php caches (apcu, opcache, ...) will be lost during httpd server restart, ...
php-fpm is part of PHP, which means that the version of fpm is the same as the PHP version.
When using a thread based httpd MPM, php can't be used as a module, unless you use ZTS, which has a whole set of painful side effects - "opening a can of worms" comes to mind.

Any sane server environment these days uses a thread based model, h2, and php-fpm (and maybe a reverse proxy somwhere in the mix). Everything else is less performant, more complicated to maintain, and less secure.

As a side note, I find it really disconcerting that h2 is still a problem and that you've decided to deactivate it again instead of making it the highest priority. There are server admins out there who have disabled http/1.0 and http/1.1 completely. The reasons should be obvious, but I will not start a h2 promotion thread here.
I'm already at the lookout for a stable 3rd party client that uses WebDAV and h2 for syncing. All the other features are useless to me anyway, since I am not using macOS Finder (so I can't right-click for the share context menu), nor am I interested in VFS. And to be honest I don't trust the newer ownCloud or Nextcloud clients enough to keep my data safe. (Please forgive me, but that's my opinion.)

P.S.: I'm more than willing to re-evaluate the ownCloud client again in the future. I'm not saying that I'm gonna turn my back on it forever. It's just that the current situation doesn't create much confidence.

@dipyalov

This comment has been minimized.

Copy link

@dipyalov dipyalov commented Dec 10, 2019

Have the same issue after 2.6.0
Plus all of my virtual files are gone and I'm worried that ownCloud may decide to sync-delete them from server.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.