Download of big file fails and leaves broken file behind.

[dragotin]

I have a big file with the size of 4656726016 bytes. It can be successfully synced up to ownCloud by the (fixed) ownCloud client.

When I do a GET request on the same file, the download stops after 361758720 bytes. The server responds with response code 200. The response body of the GET request which is sent by the server shows the correct file size of 4656726016 bytes. The client reports a "connection was closed by the server".

Another file with size 1384909928 bytes syncs fine. I tried on current master and stable45 branch.

[dragotin]

Maybe its a 2GB file size limit problem, however this test was done in a 64bit environment which should be capable. Probably the way to fix it is to not allow GETs of files >2GB by responding with a 413 "Request Entity Too Large" and have the clients using ranged downloads.

[jancborchardt]

I’m closing this bug because it has been inactive for a few months. Please reopen if the error still persists with ownCloud 5.0.9.

Thanks homie!

[steffen-moser]

I am currently experiencing the problem that downloading files larger than 2 GB fails when using the webdav access (via "https:///owncloud/remote.php/webdav/") or the sync client. In the case of the sync client, both, version 1.3.0 and the recent beta version seem to be affected.

The ownCloud server is running the most recent version (5.0.10) on a pure 64 bit test system (Ubuntu 13.04, 64 bit). I've made the same experience with our production server which runs the same ownCloud version on Solaris 11.1 (x86_64) with self-compiled PHP 5.4.18 to have a pure 64 bit chain of Apache, PHP and MySQL (Solaris 11.1 x86_64 itself still comes only with 32 bit packages of PHP).

The problem shows up in the following way: While uploading and downloading large files (I tested it up to 16 GB per file) via ownCloud's web frontend using Firefox works flawlessly on both servers (Solaris production server and Ubuntu test server), there still seems to be a limit at 2 GB as soon as the webdav access or the sync clients are getting involved. Files larger than 2 GB are corrupted, downloaded partially or the webdav connection freezes. While it's freezing, ownCloud copies the large file partially to the "/tmp" folder but the webdav client won't receive anything. At least from my perspective, there are still problems with handling files larger than 2 GB on pure 64 bit systems via webdav or via the sync client which need in-depth debugging.

Therefore I'd suggest to re-open this bug.

Can anybody else confirm this also with the recent version of ownCloud? Do I do anything wrong? Is it working flawlessly for other users?

[steffen-moser]

ownCloud-5.0.11RC1 and sync client 1.4.0 are also affected.

Uploading large files (tested up to 16 GB) via web, via webDAV and via sync works. Downloading them works only via web, but not with using webDAV or the sync client. I've been looking into this and found that the sync client really uses the webDAV access. So webDAV is the common source for the problem.

After spending some more time into debugging, I've found that the following patch seems to fix it for me:

--- owncloud/3rdparty/Sabre/HTTP/Response.php   2013-08-28 14:18:14.000000000 +0200
+++ new/3rdparty/Sabre/HTTP/Response.php        2013-09-10 09:56:32.674236315 +0200
@@ -159,7 +159,10 @@ class Sabre_HTTP_Response {

         if (is_resource($body)) {

-            fpassthru($body);
+            while (!feof($body)) {
+                echo fread($body, 8192);
+            }
+            fclose($body);

         } else {

I am not an expert in PHP programming and I've only tested it on our Solaris 11.1 server (running Apache/2.2.24 (64 bit), PHP-5.4.18 (64 bit), MySQL 5.1.37 (64 bit)) so far. It has been particularily not tested for any regressions on 32 bit servers, yet.

So maybe one of the ownCloud developers could have a look into this.

[DeepDiver1975]

actually this is related to SabreDAV @evert can I ask you for your opinion? THX

[evert]

fpassthru should just work... so if it doesn't, it would be good to somehow get a reproducable test case for this

[steffen-moser]

I think I can provide you detailed step-by-step instructions to reproduce this behavior.

Let me summarize the problem:

On a pure 64 bit server system (Linux Ubuntu 13.04 x86_64, Oracle Solaris 11.1 x86_64) downloading files larger than 2 GB can fail (download stops, files get corrupted) when downloading the files by using ownCloud's SabreDAV access (e.g. via web browser, via webDAV mount or via the SYNC Client). Uploading and Downloading of the large files doesn't cause problems when doing that via ownCloud's web interface.

Detailed steps to reproduce the problem:

1. Do a base installation of a recent Linux distribution (e.g. Ubuntu 13.04), choose, of course, the x86_64 edition.
2. Install the Apache web browser that comes with the distribution, but don't install the PHP version which is packaged by the distributor. I installed the following packages after a standard installation of Ubuntu (and all of its dependencies):

apache2-mpm-prefork
apache2-prefork-dev
libxml2-dev
libpng12-dev

1. Download a recent version of PHP (vanilla sources) from PHP.net. I've chosen version 5.4.20 for this set-up, but 5.5.4 shows the same problem.
2. Untar PHP, configure it, compile it (make) and install it (sudo make installI). I configured it with the following options:

configure --with-zlib --with-gd --enable-mbstring --enable-zip --with-pear --with-mysql --with-apxs2=/usr/bin/apxs2

1. Configure a handler in Apache for PHP in "/etc/apache2/sites-enabled/000-default/":

AddHandler php5-script .php

1. Change the AllowOverride option in the same file from "None" to "All"
2. Restart Apache
3. Download ownCloud 5.0.11 (don't install the version that has been packaged by the Linux distributor) and extract the tar file to "/var/www/owncloud". Give the web server user (www-data) write access to the necessary directories.
4. Access ownCloud with a web browser, create the admin account and change the max. file size to 20 GB.
5. Create some test files (e.g. using "dd") with sizes, for example, of 512 MB, 1 GB, 2 GB, 3 GB, 4 GB, ..., 8 GB.
6. Upload these files via ownCloud's web frontend. Uploading these files should work without any problems. After then download the files via ownClouds's web frontend. This should also work without problems and the downloaded files will not be corrupted (you can compare them to the original ones).
7. Now try to download the files again, but this time by using the SabreDAV client which comes with ownCloud. Therefore enter the following URL: http:///owncloud/remote.php/webdav

The files with 512 MB and 1 GB will be downloadable without any problems. The files with 2 GB and 3 GB won't be downloadable (connection reset by Apache), files larger than 4 GB will be downloadable but the downloaded files will be corrupted (by being downloaded only partially). As the ownCloud SYNC client also uses the SabreDAV interface, it'll run into the same problem. The problem also occurs if someone mounts the SabreDAV URL in the operating system.

The problem does not occur if you use a version of PHP which has been treated with the patch which resulted from these bug reports:

https://bugs.php.net/bug.php?id=52102
https://bugs.launchpad.net/ubuntu/+source/php5/+bug/564920

This patch is included in recent Linux distributions (e.g. Ubuntu 13.04). Therefore, the problem does not occur to the majority of ownCloud users as long as they use patched versions of PHP. We run ownCloud on a Oracle Solaris 11.1 server (on a x86_64 platform) as a host for ownCloud and other web applications. While Solaris 11.1 has a 64 bit kernel and comes with 64 bit versions of Apache and MySQL, it does only contain 32 bit packages of PHP (and the rather old version 5.3.14). To enable the handling of very large files with ownCloud, we built IPS packages for Solaris 11.1 containing 64 bit binaries of the latest PHP 5.4.x by ourselves. After installing these packages, handling large files via ownCloud's web frontend has started to work like a charm, but the SYNC client and webDAV have still been making trouble.

Possible solutions: When I apply the above given patch to "owncloud/3rdparty/Sabre/HTTP/Response.php", the problem will be fixed (at least it seems to be fixed to me). Alternative solution for us: We patch PHP by ourselves like Linux distributors do.

In my opinion, ownCloud should be able to handle very large files at least in a pure 64 bit environment without the need of Linux distributors' patches, but also with a vanilla PHP interpreter.

[bantu]

Albeit a PHP bug, the workaround ideally should be upstreamed to SabreDAV. @evert Can you tell us whether this has happened and/or is wanted?

[evert]

Hey guys.. I just tried to make a small reproduction script, and surprisingly, I could almost immediately reproduce it. So I am looking into this.

[evert]

Did some more testing:

http://evertpot.com/fpassthru-broken/

Don't think I'll learn anything new there, but I will provide a fix for sabredav shortly as well.

[roha4000]

We are experiencing the same problem here on RedHat using PHP 5.3.8 from iuscommunity. The work around from steffen-moser solved the problem. Thanks for that!

[bantu]

I am closing this in favour of the newer ticket #8578. Also see #10146 for a PR fixing this.