owncloud crashs external file system when several users download same large public link shared file, using object storage and 4 servers in the cluster

[joeyjy]

Steps to reproduce

1. let 10 users download one file(2 GB) in public link
   2.
   3.

Expected behaviour

only one temp cached file will be download on the server

Actual behaviour

10 files were downloaded from object storage, it means one download request will touch one php temp file on the webserver

Server configuration

Operating system: centos 7

Web server: apache

Database: mysql

PHP version: 5.4

ownCloud version: (see ownCloud admin page) 9.0.1

Updated from an older ownCloud or fresh install: fresh

Where did you install ownCloud from:

Signing status (ownCloud 9.0 and above):

Login as admin user into your ownCloud and access 
http://example.com/index.php/settings/integrity/failed 
paste the results here.

List of activated apps:

If you have access to your command line run e.g.:
sudo -u www-data php occ app:list
from within your ownCloud installation folder

The content of config/config.php:

If you have access to your command line run e.g.:
sudo -u www-data php occ config:list system
from within your ownCloud installation folder

or 

Insert your config.php content here
(Without the database password, passwordsalt and secret)

Are you using external storage, if yes which one: local/smb/sftp/... ceph object storage

Are you using encryption: yes/no yes

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/... ldap

LDAP configuration (delete this part if not used)

With access to your command line run e.g.:
sudo -u www-data php occ ldap:show-config
from within your ownCloud installation folder

Without access to your command line download the data/owncloud.db to your local
computer or access your SQL server remotely and run the select query:
SELECT * FROM `oc_appconfig` WHERE `appid` = 'user_ldap';


Eventually replace sensitive data as the name/IP-address of your LDAP server or groups.

Client configuration

Browser:

Operating system:

Logs

Web server error log

Insert your webserver log here

ownCloud log (data/owncloud.log)

Insert your ownCloud log here

Browser log

Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log 
c) ...

Hi OC team,

Can we cache only one file on the server when download same file? I found when several users download one same file in the public link, each request will make web server(4 servers in one cluster) download the file from object storage, so if there are 10 users downloading one same file at the same time, there would be 10 files on my 4 webservers, it will crash my backend object storage

need your help and advisor, thanks!

[PVince81]

This means that several PHP requests would need to be able to access the same temporarily downloaded file, which is tricky considering that there also needs to be some kind of lock / tracking to make sure subsequent requests wait for the first request to fully finish downloading the file.

This comes closer to what I suggested here: #5225

On the other hand, I think that a temporary file shouldn't even be needed and the file should be streamable directly from the swift storage, which is discussed here: #5949

From what I see SWIFT stream download has already been implemented in 9.0 here #19002, so the question is why is it still creating temporary files in your specific environment ?

[joeyjy]

@PVince81 Thanks for reply! I use object storage as the primary storage backend rather than files_external and enabled encryption, if SWIFT stream download also works in my situation?

[PVince81]

Looks like primary swift storage should also use streaming:

• https://github.com/owncloud/core/blob/v9.0.4/lib/private/files/objectstore/swift.php#L131
• https://github.com/owncloud/core/blob/v9.0.4/lib/private/files/objectstore/objectstorestorage.php#L258

CC @butonic

[joeyjy]

@PVince81 Thanks! If I want to cache some large file in web server, where can I modify in the source code? I mean every time a large file download request came, I can download the origin file from object storage and keep it on the web server for a period of time in case of someone else also want to download it, I can use redis to lock that file

[PVince81]

You can't do this easily without modifying/rewriting big parts of the code unfortunately.

[joeyjy]

@PVince81 I rewrote fopen function and now file can be cached to local server, close this issue

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.