Error when running file scan on Owncloud instance with objectstore primary storage backend

[rontrompert]

Steps to reproduce

1. Run a file scan on a Owncloud instance with an S3 object store storage backend as primary storage.

Expected behaviour

file scan runs normally

Actual behaviour

Get an error message:
root# sudo -u apache php occ files:scan ron
Starting scan for user 1 out of 1 (ron)
Exception during scan: Undefined index: size
#0 /var/www/owncloud10/lib/private/Files/Cache/Scanner.php(421): OCA\Files\Command\Scan->exceptionErrorHandler(8, 'Undefined index...', '/var/www/ownclo...', 421, Array)
#1 /var/www/owncloud10/lib/private/Files/Cache/Scanner.php(382): OC\Files\Cache\Scanner->handleChildren('', true, 3, '12977', true, 16102413)
#2 /var/www/owncloud10/lib/private/Files/Cache/Scanner.php(316): OC\Files\Cache\Scanner->scanChildren('', true, 3, '12977', true)
#3 /var/www/owncloud10/lib/private/Files/Utils/Scanner.php(235): OC\Files\Cache\Scanner->scan('', true, 3)
#4 /var/www/owncloud10/apps/files/lib/Command/Scan.php(159): OC\Files\Utils\Scanner->scan('/ron')
#5 /var/www/owncloud10/apps/files/lib/Command/Scan.php(228): OCA\Files\Command\Scan->scanFiles('ron', '/ron', false, Object(Symfony\Component\Console\Output\ConsoleOutput), false)
#6 /var/www/owncloud10/lib/composer/symfony/console/Command/Command.php(262): OCA\Files\Command\Scan->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#7 /var/www/owncloud10/core/Command/Base.php(159): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#8 /var/www/owncloud10/lib/composer/symfony/console/Application.php(826): OC\Core\Command\Base->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#9 /var/www/owncloud10/lib/composer/symfony/console/Application.php(189): Symfony\Component\Console\Application->doRunCommand(Object(OCA\Files\Command\Scan), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#10 /var/www/owncloud10/lib/composer/symfony/console/Application.php(120): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#11 /var/www/owncloud10/lib/private/Console/Application.php(160): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#12 /var/www/owncloud10/console.php(106): OC\Console\Application->run()
#13 /var/www/owncloud10/occ(11): require_once('/var/www/ownclo...')
#14 {main}

Server configuration

Operating system:
CentOS7.3

Web server:
apache2.4.6

Database:
mariadb-5.5

PHP version:
7.0

ownCloud version: (see ownCloud admin page)
10.0.3 enterprise

Updated from an older ownCloud or fresh install:
Fresh install

Where did you install ownCloud from:
tar ball

Signing status (ownCloud 9.0 and above):

Login as admin user into your ownCloud and access 
http://example.com/index.php/settings/integrity/failed 
paste the results into https://gist.github.com/ and puth the link here.

No errors have been found.

The content of config/config.php:

Log in to the web-UI with an administrator account and click on
'admin' -> 'Generate Config Report' -> 'Download ownCloud config report'
This report includes the config.php settings, the list of activated apps
and other details in a well sanitized form.

or 

If you have access to your command line run e.g.:
sudo -u www-data php occ config:list system
from within your ownCloud installation folder

*ATTENTION:* Do not post your config.php file in public as is. Please use one of the above
methods whenever possible. Both, the generated reports from the web-ui and from occ config:list
consistently remove sensitive data. You still may want to review the report before sending.
If done manually then it is critical for your own privacy to dilligently
remove *all* host names, passwords, usernames, salts and other credentials before posting.
You should assume that attackers find such information and will use them against your systems.

{
"system": {
"instanceid": "oco1leooohkc",
"passwordsalt": "REMOVED SENSITIVE VALUE",
"secret": "REMOVED SENSITIVE VALUE",
"trusted_domains": [
"test.data.surfsara.nl"
],
"license-key": "REMOVED SENSITIVE VALUE",
"overwrite.cli.url": "https://test.data.surfsara.nl",
"overwritehost": "test.data.surfsara.nl",
"overwriteprotocol": "https",
"forwarded_for_headers": [
"HTTP_X_FORWARDED_FOR",
"HTTP_FORWARDED_FOR"
],
"memcache.local": "\OC\Memcache\APCu",
"memcache.distributed": "\OC\Memcache\Redis",
"memcache.locking": "\OC\Memcache\Redis",
"redis": {
"host": "145.100.5.230",
"port": 6379,
"timeout": 0,
"password": "REMOVED SENSITIVE VALUE",
"dbindex": 0
},
"filelocking.enabled": "true",
"trusted_proxies": [
"145.100.5.230"
],
"logfile": "/var/www/owncloud10/data/owncloud.log",
"dbtype": "mysql",
"version": "10.0.3.3",
"dbname": "owncloud10",
"dbhost": "145.100.5.230",
"dbtableprefix": "oc_",
"dbuser": "REMOVED SENSITIVE VALUE",
"dbpassword": "REMOVED SENSITIVE VALUE",
"logtimezone": "UTC",
"installed": true,
"objectstore_multibucket": {
"class": "OCA\ObjectStore\S3",
"arguments": {
"autocreate": true,
"options": {
"version": "2006-03-01",
"region": "",
"credentials": {
"key": "REMOVED SENSITIVE VALUE",
"secret": "REMOVED SENSITIVE VALUE"
},
"endpoint": "https://proxy.swift.surfsara.nl:443",
"command.params": {
"PathStyle": true
}
}
}
}
}
}

List of activated apps:

If you have access to your command line run e.g.:
sudo -u www-data php occ app:list
from within your ownCloud installation folder.

admin_audit: 0.8.1
comments: 0.3.0
configreport: 0.1.1
dav: 0.3.0
enterprise_key: 0.1.3
federatedfilesharing: 0.3.1
federation: 0.1.0
files: 1.5.1
files_external: 0.7.1
files_sharing: 0.10.1
files_trashbin: 0.9.1
files_versions: 1.3.0
files_videoplayer: 0.9.8
firewall: 2.4.2
firstrunwizard: 1.1
market: 0.2.2
notifications: 0.3.1
objectstore: 0.4
provisioning_api: 0.5.0
systemtags: 0.3.0
systemtags_management: 0.2.0
templateeditor: 0.1
theme-enterprise: 2.0.0
updatenotification: 0.2.1
windows_network_drive: 0.6.1
workflow: 0.2.5
Disabled:
encryption
external
files_antivirus
files_ldap_home
guests
password_policy
sharepoint
theme-example
user_external
user_shibboleth

Are you using external storage, if yes which one: local/smb/sftp/...
No

Are you using encryption: yes/no
No

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
No

LDAP configuration (delete this part if not used)

With access to your command line run e.g.:
sudo -u www-data php occ ldap:show-config
from within your ownCloud installation folder

Without access to your command line download the data/owncloud.db to your local
computer or access your SQL server remotely and run the select query:
SELECT * FROM `oc_appconfig` WHERE `appid` = 'user_ldap';


Eventually replace sensitive data as the name/IP-address of your LDAP server or groups.

Client configuration

Browser:
Firefox
Operating system:

Logs

Web server error log

Insert your webserver log here

ownCloud log (data/owncloud.log)

Insert your ownCloud log here

Browser log

Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log 
c) ...

[PVince81]

Files scan doesn't make sense when using primary object store: the object store only contains a flat list of file ids as keys and file contents as objects, so there is nothing useful to scan.

Please note that you cannot use primary object store mode with ownCloud if you uploaded files there using the usual S3/Swift uploaders that would use absolute paths as keys.

cc @DeepDiver1975

[rontrompert]

Using a file system we have seen in practice that situations can occur that there at some point an inconsistencies can develop between database and storage system for whatever reason. This is understandable because writing stuff to a database and write the stuff to storage is not an atomic action and at each stage things can fail.

On a normal file system the file scan can help here to bring things into sync. But there is no equivalent for an object store.

[SURFsara-e-infra]

Another thing is that in a disaster recovery scenario where there are separate backups of database and storage, the database is by definition out of sync with the storage.

[PVince81]

I'll reopen this for discussion but I don't think it is possible to rescan a primary object store as the scanner cannot find out what to put into oc_filecache, beacause all objects inside the object store are mapped by file id and file contents. There isn't any other information there. So by scanning, the scanner cannot know what to do with fileid + contents: no way to find in what path this object is supposed to be.

Now if we do consider recovery scenarios where the backups aren't from the same moment, it would require storing more metadata into the object store like file path, etc.

@DeepDiver1975 @butonic

[TrurlMcByte]

I see additional problem in it, during scan external S3-like storage error happens on first subdirectory, but verbose output show this folder as "File"

UPD: S3 don't require to create "folders" (also s3fs not compatible with "folders" at all, it's work only with files-with-full-path-inside). When "folders" (as empty objects) is created files:scan have no errors in this case