Large external storage deletions can fill data drive

[dbchelgit]

Steps to reproduce

1. Put a bunch of large files in external storage (eg, S3) where the size exceeds the space available in the Nextcloud data filesystem
2. Make sure trash bin is enabled
3. Remove the files

Expected behaviour

Hmm. Well, one could argue that either the files should fail to be thrown out, an alert should be generated, with a "Yes, proceed" dialog, or something else? But...it probably shouldn't wedge the server.

Actual behaviour

Nextcloud copies everything into the trash bin until space runs out on the filesystem. Server is then wedged and requires manual intervention (SSH in to clean stuff) and a reboot or restart.

General server configuration

Operating system: Linux owncloud 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20 20:32:50 UTC 2017 x86_64

Web server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.0.24 (apache2handler)

Database: mysql 10.1.19

PHP version: 7.0.24

PHP-modules loaded

 - Core
 - date
 - libxml
 - openssl
 - pcre
 - zlib
 - filter
 - hash
 - Reflection
 - SPL
 - session
 - standard
 - apache2handler
 - bz2
 - calendar
 - ctype
 - curl
 - dom
 - mbstring
 - fileinfo
 - ftp
 - gd
 - gettext
 - iconv
 - intl
 - json
 - exif
 - mcrypt
 - mysqlnd
 - PDO
 - Phar
 - posix
 - shmop
 - SimpleXML
 - sockets
 - sqlite3
 - sysvmsg
 - sysvsem
 - sysvshm
 - tokenizer
 - xml
 - xmlwriter
 - xsl
 - zip
 - mysqli
 - pdo_mysql
 - pdo_sqlite
 - wddx
 - xmlreader
 - apcu
 - Zend OPcache

Nextcloud configuration

Nextcloud version: 12.0.3 - 12.0.3.3

**Updated from an older Nextcloud/ownCloud or fresh install: Was upgraded

Where did you install Nextcloud from: Downloaded from Nextcloud directly

Are you using external storage, if yes which one: Array
(
[0] => \OC\Files\Storage\Local
[1] => \OCA\Files_External\Lib\Storage\FTP
[2] => \OC\Files\Storage\DAV
[3] => \OCA\Files_External\Lib\Storage\OwnCloud
[4] => \OCA\Files_External\Lib\Storage\SFTP
[5] => \OCA\Files_External\Lib\Storage\AmazonS3
[6] => \OCA\Files_External\Lib\Storage\Dropbox
[7] => \OCA\Files_External\Lib\Storage\Google
[8] => \OCA\Files_External\Lib\Storage\Swift
[9] => \OCA\Files_External\Lib\Storage\SFTP
)

Are you using encryption: no

Are you using an external user-backend, if yes which one: YOUR ANSWER HERE (LDAP/ActiveDirectory/Webdav/...)

Signing status

[]

Enabled apps

 - activity: 2.5.2
 - bruteforcesettings: 1.0.2
 - comments: 1.2.0
 - dav: 1.3.0
 - federatedfilesharing: 1.2.0
 - federation: 1.2.0
 - files: 1.7.2
 - files_external: 1.3.0
 - files_pdfviewer: 1.1.1
 - files_sharing: 1.4.0
 - files_texteditor: 2.4.1
 - files_trashbin: 1.2.0
 - files_versions: 1.5.0
 - files_videoplayer: 1.1.0
 - firstrunwizard: 2.1
 - gallery: 17.0.0
 - issuetemplate: 0.2.2
 - logreader: 2.0.0
 - lookup_server_connector: 1.0.0
 - nextcloud_announcements: 1.1
 - notifications: 2.0.0
 - oauth2: 1.0.5
 - password_policy: 1.2.2
 - provisioning_api: 1.2.0
 - serverinfo: 1.2.0
 - sharebymail: 1.2.0
 - survey_client: 1.0.0
 - systemtags: 1.2.0
 - theming: 1.3.0
 - twofactor_backupcodes: 1.1.1
 - updatenotification: 1.2.0
 - workflowengine: 1.2.0

Disabled apps

 - admin_audit
 - encryption
 - user_external
 - user_ldap

Content of config/config.php

{
    "instanceid": "ocl1styuwl51",
    "passwordsalt": "***REMOVED SENSITIVE VALUE***",
    "secret": "***REMOVED SENSITIVE VALUE***",
    "trusted_domains": [
        "**REMOVED**"
    ],
    "datadirectory": "\/var\/www\/html\/nextcloud\/data",
    "overwrite.cli.url": "**REMOVED**",
    "dbtype": "mysql",
    "version": "12.0.3.3",
    "dbname": "nextcloud",
    "dbhost": "***REMOVED***",
    "dbport": "3306",
    "dbtableprefix": "oc_",
    "dbuser": "***REMOVED SENSITIVE VALUE***",
    "dbpassword": "***REMOVED SENSITIVE VALUE***",
    "logtimezone": "UTC",
    "installed": true,
    "memcache.local": "\\OC\\Memcache\\APCu",
    "updater.release.channel": "stable",
    "mail_from_address": "owncloud",
    "mail_smtpmode": "smtp",
    "mail_domain": "**REMOVED*",
    "mail_smtpauthtype": "PLAIN",
    "mail_smtpauth": 1,
    "mail_smtphost": "**REMOVED**",
    "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
    "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpport": "465",
    "mail_smtpsecure": "ssl",
    "theme": "",
    "loglevel": 2,
    "maintenance": false
}

Client configuration

Browser: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/604.3.5 (KHTML, like Gecko) Version/11.0.1 Safari/604.3.5

Operating system: CentOS

Logs

Web server error log

Insert your webserver log here

Nextcloud log (data/nextcloud.log)

Insert your Nextcloud log here

Browser log

Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log
c) ...

[blizzz]

@schiessle

[dbchelgit]

Further investigation here shows that my trash and version file deletion doesn't seem to be working at all...at least, over the space of a few days the filesystem can still fill up (eg, the 50% and other checks aren't kicking in). Any way to debug why this is happening? I don't see any selinux or other complaints about an unlink fails. I do see this:

type=AVC msg=audit(1516401061.514:11320): avc: denied { getattr } for pid=23973 comm="find" path="/usr/sbin/crond" dev="xvda1" ino=395861 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:crond_exec_t:s0 tclass=file

But it's not happening every 15 minutes, so doesn't seem like a find failing to look for old files.

[jdinkel]

I think expected behavior is file deletions on external storage would go to a trashbin on the external storage. I think simplest would be to let the user/admin define a location for the trash when creating the external storage.

[shukriadams]

Just hit this, VM drive got filled to 100% while I was making changes to files in S3. No idea how I'm going to unclog the mess this made.

[szaimen]

Hi, I think this is rather a feature request to allow an external direct delete or allowing to create a hidden trash folder on the external storage directly.

[joshtrichards]

Duplicate of #14436

Let's consolidate discussions related to this there.