occ files:scan bails out with “StorageNotAvailableException” #18929

4001982248998 · 2015-09-09T13:31:33Z

while performing a filescan, the process bails out (always at the same file) with this notice:

  [OCP\Files\StorageNotAvailableException]  

files:scan [-p|--path="..."] [-q|--quiet] [--all] [user_id1] ... [user_idN]

It does not give any more hints. What can be the cause, and what could be done to correct it?

The text was updated successfully, but these errors were encountered:

LukasReschke · 2015-09-09T15:14:12Z

Please use the issue template as described at https://github.com/owncloud/core/blob/master/CONTRIBUTING.md, thx.

4001982248998 · 2015-09-10T11:47:36Z

@LukasReschke and everybody else: I am really sorry! Here is the info.

Steps to reproduce

using the owncloud console, issue the command to scan all files.

user@host $ sudo su -s /bin/bash www-data
www-data@host $ owncloud/occ files:scan --all

Expected behaviour

It should scan all files in the file system and update the database accordingly.

Actual behaviour

The occ command does not complete, does display the following error message (in large red borders):

  [OCP\Files\StorageNotAvailableException]  

files:scan [-p|--path="..."] [-q|--quiet] [--all] [user_id1] ... [user_idN]

Server configuration

Operating system: Ubuntu 14.04.3 LTS, 64 Bit

Web server: Apache2 2.4.7-1ubuntu4.5

Database: MySQL 5.5.44-0ubuntu0.14.04.1

PHP version: PHP 5.5.9+dfsg-1ubuntu4.11

ownCloud version: 8.1.1

Updated from an older ownCloud or fresh install:
updated in several steps, from initially version 4.x to lastly from 7.0.4 to 8.1.1

List of activated apps:

Enabled:
 - activity: 2.0.1
 - files: 1.1.10
 - files_pdfviewer: 0.7
 - files_sharing: 0.6.2
 - files_texteditor: 0.4
 - files_trashbin: 0.6.3
 - files_versions: 1.0.6
 - files_videoviewer: 0.1.3
 - firstrunwizard: 1.1
 - gallery: 0.6.0
 - user_external: 0.4
 - user_ldap: 0.6.1
Disabled:
 - encryption
 - external
 - files_external
 - files_locking
 - provisioning_api
 - templateeditor
 - user_webdavauth

The content of config/config.php:

<?php
$CONFIG = array (
  'instanceid' => '514c1e72ae374',
  'passwordsalt' => 'passwordsalt',
  'datadirectory' => '/var/www/vhost/oc5/public/data',
  'dbtype' => 'mysql',
  'version' => '8.1.1.3',
  'dbname' => 'owncloud2013',
  'dbhost' => '127.0.0.1',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'oc_root',
  'dbpassword' => 'dbpassword',
  'installed' => true,
  'ldapIgnoreNamingRules' => false,
  'log_type' => 'syslog',
  'logfile' => '',
  'loglevel' => '2',
  'maintenance' => false,
  'theme' => '',
  'trusted_domains' => 
  array (
    0 => 'ahostname',
    1 => 'anotherhostname',
  ),
  'secret' => 'thatsasecret,
  'share_folder' => '/Shared',
);

Are you using external storage, if yes which one: local/smb/sftp/...
no

Are you using encryption: yes/no
no

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
yes, two LDAP backends:

MS AD
OpenLDAP

LDAP configuration (delete this part if not used)

+-------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|�[32m Configuration                 �[39m|�[32m                                                                                                                                                                                                                          �[39m|
+-------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| hasMemberOfFilterSupport      | 1                                                                                                                                                                                                                        |
| hasPagedResultSupport         |                                                                                                                                                                                                                          |
| homeFolderNamingRule          | attr:userPrincipalName                                                                                                                                                                                                   |
| lastJpegPhotoLookup           | 0                                                                                                                                                                                                                        |
| ldapAgentName                 | CN=ldapreaderowncloud,OU=OwnCloud,[…]
| ldapAgentPassword             | ***                                                                                                                                                                                                                      |
| ldapAttributesForGroupSearch  |                                                                                                                                                                                                                          |
| ldapAttributesForUserSearch   |                                                                                                                                                                                                                          |
| ldapBackupHost                |                                                                                                                                                                                                                          |
| ldapBackupPort                | 389                                                                                                                                                                                                                      |
| ldapBase                      | […]
| ldapBaseGroups                | […]
| ldapBaseUsers                 | […]
| ldapCacheTTL                  | 600                                                                                                                                                                                                                      |
| ldapConfigurationActive       | 1                                                                                                                                                                                                                        |
| ldapEmailAttribute            | mail                                                                                                                                                                                                                     |
| ldapExperiencedAdmin          | 0                                                                                                                                                                                                                        |
| ldapExpertUUIDGroupAttr       |                                                                                                                                                                                                                          |
| ldapExpertUUIDUserAttr        |                                                                                                                                                                                                                          |
| ldapExpertUsernameAttr        |                                                                                                                                                                                                                          |
| ldapGroupDisplayName          | cn                                                                                                                                                                                                                       |
| ldapGroupFilter               |                                                                                                                                                                                                                          |
| ldapGroupFilterGroups         |                                                                                                                                                                                                                          |
| ldapGroupFilterMode           | 0                                                                                                                                                                                                                        |
| ldapGroupFilterObjectclass    |                                                                                                                                                                                                                          |
| ldapGroupMemberAssocAttr      | member                                                                                                                                                                                                                   |
| ldapHost                      | […]
| ldapIgnoreNamingRules         |                                                                                                                                                                                                                          |
| ldapLoginFilter               | (&(memberOf:1.2.840.113556.1.4.1941:=CN=FNF-PRO-OWNCLOUD,[…]
| ldapLoginFilterAttributes     |                                                                                                                                                                                                                          |
| ldapLoginFilterEmail          | 0                                                                                                                                                                                                                        |
| ldapLoginFilterMode           | 1                                                                                                                                                                                                                        |
| ldapLoginFilterUsername       | 0                                                                                                                                                                                                                        |
| ldapNestedGroups              | 0                                                                                                                                                                                                                        |
| ldapNoCase                    | 1                                                                                                                                                                                                                        |
| ldapOverrideMainServer        | 0                                                                                                                                                                                                                        |
| ldapPagingSize                | 500                                                                                                                                                                                                                      |
| ldapPort                      | 389                                                                                                                                                                                                                      |
| ldapQuotaAttribute            | Pager                                                                                                                                                                                                                    |
| ldapQuotaDefault              | 5368709120                                                                                                                                                                                                               |
| ldapTLS                       | 1                                                                                                                                                                                                                        |
| ldapUserDisplayName           | cn                                                                                                                                                                                                                       |
| ldapUserFilter                | […]
| ldapUserFilterGroups          | […]
| ldapUserFilterMode            | 1                                                                                                                                                                                                                        |
| ldapUserFilterObjectclass     |                                                                                                                                                                                                                          |
| ldapUuidGroupAttribute        | auto                                                                                                                                                                                                                     |
| ldapUuidUserAttribute         | auto                                                                                                                                                                                                                     |
| turnOffCertCheck              | 0                                                                                                                                                                                                                        |
| useMemberOfToDetectMembership | 1                                                                                                                                                                                                                        |
+-------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+-------------------------------+-------------------------------------------------------------------------------------------------------------------------+
|�[32m Configuration                 �[39m|�[32m s01                                                                                                                     �[39m|
+-------------------------------+-------------------------------------------------------------------------------------------------------------------------+
| hasMemberOfFilterSupport      | 1                                                                                                                       |
| hasPagedResultSupport         |                                                                                                                         |
| homeFolderNamingRule          | attr:cn                                                                                                                 |
| lastJpegPhotoLookup           | 0                                                                                                                       |
| ldapAgentName                 |                                                                                                                         |
| ldapAgentPassword             | ***                                                                                                                     |
| ldapAttributesForGroupSearch  |                                                                                                                         |
| ldapAttributesForUserSearch   |                                                                                                                         |
| ldapBackupHost                |                                                                                                                         |
| ldapBackupPort                | 389                                                                                                                     |
| ldapBase                      | […]
| ldapBaseGroups                | […]
| ldapBaseUsers                 | […]
| ldapCacheTTL                  | 600                                                                                                                     |
| ldapConfigurationActive       | 1                                                                                                                       |
| ldapEmailAttribute            | mail                                                                                                                    |
| ldapExperiencedAdmin          | 0                                                                                                                       |
| ldapExpertUUIDGroupAttr       |                                                                                                                         |
| ldapExpertUUIDUserAttr        |                                                                                                                         |
| ldapExpertUsernameAttr        |                                                                                                                         |
| ldapGroupDisplayName          | cn                                                                                                                      |
| ldapGroupFilter               |                                                                                                                         |
| ldapGroupFilterGroups         |                                                                                                                         |
| ldapGroupFilterMode           | 0                                                                                                                       |
| ldapGroupFilterObjectclass    |                                                                                                                         |
| ldapGroupMemberAssocAttr      | member                                                                                                                  |
| ldapHost                      | ldaps://[…]
| ldapIgnoreNamingRules         |                                                                                                                         |
| ldapLoginFilter               | […]
| ldapLoginFilterAttributes     | cn                                                                                                                      |
| ldapLoginFilterEmail          | 0                                                                                                                       |
| ldapLoginFilterMode           | 1                                                                                                                       |
| ldapLoginFilterUsername       | 1                                                                                                                       |
| ldapNestedGroups              | 0                                                                                                                       |
| ldapNoCase                    |                                                                                                                         |
| ldapOverrideMainServer        |                                                                                                                         |
| ldapPagingSize                | 500                                                                                                                     |
| ldapPort                      | 636                                                                                                                     |
| ldapQuotaAttribute            | host                                                                                                                    |
| ldapQuotaDefault              | 5368709120                                                                                                              |
| ldapTLS                       |                                                                                                                         |
| ldapUserDisplayName           | displayname                                                                                                             |
| ldapUserFilter                | […]
| ldapUserFilterGroups          | Owncloud                                                                                                                |
| ldapUserFilterMode            | 1                                                                                                                       |
| ldapUserFilterObjectclass     | sambaSamAccount                                                                                                         |
| ldapUuidGroupAttribute        | auto                                                                                                                    |
| ldapUuidUserAttribute         | auto                                                                                                                    |
| turnOffCertCheck              |                                                                                                                         |
| useMemberOfToDetectMembership | 1                                                                                                                       |
+-------------------------------+-------------------------------------------------------------------------------------------------------------------------+

Client configuration

Browser:
Firefox 40.0.4

Operating system:
Arch Linux 64 Bit

Logs

Web server error log

(empty)

ownCloud log (data/owncloud.log)

syslog is used instead:

tail /var/log/syslog
Sep 10 13:33:39 cloud ownCloud[9148]: {PHP} ldap_start_tls(): Unable to start TLS: Server is unavailable at /var/www/owncloud/apps/user_ldap/lib/ldap.php#257
Sep 10 13:33:44 cloud ownCloud[5511]: message repeated 8 times: [ {PHP} ldap_start_tls(): Unable to start TLS: Server is unavailable at /var/www/owncloud/apps/user_ldap/lib/ldap.php#257]
Sep 10 13:33:44 cloud ownCloud[5511]: {PHP} filesize(): stat failed for /var/www/vhost/oc5/public/data/owncloud.log at /var/www/owncloud/settings/admin.php#49
Sep 10 13:33:44 cloud ownCloud[5511]: {PHP} ldap_start_tls(): Unable to start TLS: Server is unavailable at /var/www/owncloud/apps/user_ldap/lib/ldap.php#257
Sep 10 13:33:45 cloud ownCloud[9127]: {PHP} ldap_start_tls(): Unable to start TLS: Server is unavailable at /var/www/owncloud/apps/user_ldap/lib/ldap.php#257
Sep 10 13:33:46 cloud ownCloud[5513]: {PHP} ldap_start_tls(): Unable to start TLS: Server is unavailable at /var/www/owncloud/apps/user_ldap/lib/ldap.php#257
Sep 10 13:33:46 cloud ownCloud[5511]: message repeated 5 times: [ {PHP} ldap_start_tls(): Unable to start TLS: Server is unavailable at /var/www/owncloud/apps/user_ldap/lib/ldap.php#257]
Sep 10 13:33:46 cloud ownCloud[5511]: {PHP} ldap_read(): Search: Invalid DN syntax at /var/www/owncloud/apps/user_ldap/lib/ldap.php#257
Sep 10 13:33:46 cloud ownCloud[5511]: {PHP} ldap_start_tls(): Unable to start TLS: Server is unavailable at /var/www/owncloud/apps/user_ldap/lib/ldap.php#257
Sep 10 13:39:01 cloud CRON[12380]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))

Browser log

(not relevant I suppose)

LukasReschke · 2015-09-10T11:49:02Z

The LDAP logs look fishy.

cc @icewind1991 @blizzz

PVince81 · 2015-09-17T09:24:16Z

In 8.2 / master there is already a fix that makes the scanner more tolerant and prevents it to bail out, see #18523

blizzz · 2015-09-25T12:37:22Z

@svenehret are you sure your first LDAP server does work at all? Appears that TLS is not enabled. Turn off ldapTLS for the first server by ./occ ldap:set-config "" ldapTLS 0 as web user to turn it off.

4001982248998 · 2015-11-27T10:28:21Z

@blizz yes, I am sure that the first LDAP server is working, because I am using an account from its directory service. It is correct that TLS is not enabled in this case.
I have tried turning off TLS like you advised, but the result with files:scan --all is exactly the same.

4001982248998 · 2015-11-27T10:30:13Z

I have re-tried with a new VM clone and with OC version 8.2.1, with the same result.
I am hereby appending an excerpt of an strace process that I was running alongside
occ-scan-excerpt.txt

PVince81 · 2016-03-09T15:30:55Z

@icewind1991 didn't we fix this recently ? Can you find the PR ?
Is it #19128 ?

PVince81 · 2016-03-09T15:32:53Z

@cdamken I also seem to remember that you were in a similar ticket, do you remember which ?

PVince81 · 2016-03-09T15:37:52Z

@svenehret is this still happening with 8.2.3 or 9.0.0 ? I seem to remember that we already had a fix for this but cannot find the PR any more.

4001982248998 · 2016-03-10T15:26:23Z

thank you for the follow-up. I have cloned the 7.0 installation stepwise to 9.0 and now it goes like this:

user@cloud:~$ occ status
  - installed: true
  - version: 9.0.0.19
  - versionstring: 9.0.0
  - edition: 
user@cloud:~$ occ files:scan --all
PHP Fatal error:  Call to a member function file_exists() on a non-object in /var/www/owncloud/lib/private/files/filesystem.php on line 693

Log:

Mar 10 16:25:44 cloud ownCloud[5195]: {PHP} Argument 7 passed to OCA\user_ldap\lib\user\Manager::__construct() must implement interface OCP\IUserManager, instance of OCA\user_ldap\lib\user\Manager given, called in /var/www/owncloud/apps/user_ldap/lib/proxy.php on line 76 and defined at /var/www/owncloud/apps/user_ldap/lib/user/manager.php#90
Mar 10 16:25:44 cloud ownCloud[5195]: {PHP} ldap_start_tls(): Unable to start TLS: Server is unavailable at /var/www/owncloud/apps/user_ldap/lib/ldap.php#264
Mar 10 16:25:45 cloud ownCloud[5195]: {PHP} Argument 9 passed to OCA\user_ldap\lib\user\User::__construct() must implement interface OCP\IUserManager, instance of OCA\user_ldap\lib\user\Manager given, called in /var/www/owncloud/apps/user_ldap/lib/user/manager.php on line 121 and defined at /var/www/owncloud/apps/user_ldap/lib/user/user.php#111
Mar 10 16:25:45 cloud ownCloud[1493]: {PHP} Argument 7 passed to OCA\user_ldap\lib\user\Manager::__construct() must implement interface OCP\IUserManager, instance of OCA\user_ldap\lib\user\Manager given, called in /var/www/owncloud/apps/user_ldap/lib/proxy.php on line 76 and defined at /var/www/owncloud/apps/user_ldap/lib/user/manager.php#90
Mar 10 16:25:45 cloud ownCloud[1493]: {PHP} ldap_start_tls(): Unable to start TLS: Server is unavailable at /var/www/owncloud/apps/user_ldap/lib/ldap.php#264
Mar 10 16:25:45 cloud ownCloud[1493]: {PHP} Argument 9 passed to OCA\user_ldap\lib\user\User::__construct() must implement interface OCP\IUserManager, instance of OCA\user_ldap\lib\user\Manager given, called in /var/www/owncloud/apps/user_ldap/lib/user/manager.php on line 121 and defined at /var/www/owncloud/apps/user_ldap/lib/user/user.php#111
Mar 10 16:25:45 cloud ownCloud[5195]: message repeated 2 times: [ {PHP} Argument 9 passed to OCA\user_ldap\lib\user\User::__construct() must implement interface OCP\IUserManager, instance of OCA\user_ldap\lib\user\Manager given, called in /var/www/owncloud/apps/user_ldap/lib/user/manager.php on line 121 and defined at /var/www/owncloud/apps/user_ldap/lib/user/user.php#111]
Mar 10 16:25:46 cloud php[5195]: {PHP} Call to a member function file_exists() on a non-object at /var/www/owncloud/lib/private/files/filesystem.php#693

alias occ='sudo -u www-data php /var/www/owncloud/occ'

No files or directories are visible in the web interface, only “Dieses Verzeichnis ist nicht verfügbar, bitte überprüfen Sie die Logdateien oder kontaktieren Sie den Administrator” in a yellow box at the top.

4001982248998 · 2016-03-10T15:38:09Z

ldap seems to work though:

user@cloud:~$ occ ldap:test-config "s01"
The configuration is valid and the connection could be established!
user@cloud:~$ occ ldap:test-config ""
The configuration is valid and the connection could be established!

I am also seeing users in the web interface, alongside LDAP backend

4001982248998 · 2016-03-14T07:57:06Z

after “some” more work I managed to upgrade in so far that occ files:scan --all finishes without error (current showstopper now are lost share data due to uncorrectible legacy storage IDs, but that's another story and shall be told elsewhere).

What I did was:

upgrade to 8.2.3 by adding the (Ubuntu) Utopic Sources to my Trusty Server
occ seemingly has better exception notifications and I was able to work around them:
- work around the error OC\ServerNotAvailableException by temporarily disabling LDAP-TLS (LDAPs lost connection, LDAP works #20672) (occ ldap:set-config "s01" ldapPort 389 ; occ ldap:set-config "s01" ldapHost ldap://server.tld)
- work around the error OCP\Lock\LockedException by manually deleting the file lock from the database (delete from oc_file_locks where oc_file_locks.key='files/c84b69a5cc0560d5b624cabeacad833e';)
- work around the error OC\User\NoUserException by deleting database rows in oc_share containing user accounts that are since deleted (handle NoUserException in sharing code #19980) (delete from oc_share where oc_share.uid_owner='149AD165-D850-4425-9C41-F1ADD946678E';)
- work around the error OC\ServerNotAvailableException (again?) by simply re-running occ files:scan --all(?)
upgrade to 9.0.0
the empty webinterface was due to Upgrade to 9 and a uiser got folder is not available (no files) #23052 and could be solved with applying the patched files (dont break when there is an invalid share #23039).

So, there's that and I'll close.

icewind1991 · 2016-03-16T13:03:04Z

unavailable storages are already skipped in the scanner since 9.0/8.2.3/8.1.6 (#20789) so I think we can close this

lock · 2019-08-05T12:01:23Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

LukasReschke added the needs info label Sep 9, 2015

LukasReschke removed the needs info label Sep 10, 2015

LukasReschke added Type:Bug triage app:files labels Sep 10, 2015

blizzz added this to the 9.0.1-next-maintenance milestone Mar 1, 2016

PVince81 added the sev2-high label Mar 9, 2016

PVince81 assigned icewind1991 Mar 9, 2016

4001982248998 closed this as completed Mar 14, 2016

cdamken mentioned this issue Jun 26, 2017

update WND Listener documentation avoiding to many jumps owncloud-archive/documentation#3180

Closed

lock bot locked as resolved and limited conversation to collaborators Aug 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

occ files:scan bails out with “StorageNotAvailableException” #18929

occ files:scan bails out with “StorageNotAvailableException” #18929

4001982248998 commented Sep 9, 2015

LukasReschke commented Sep 9, 2015

4001982248998 commented Sep 10, 2015

LukasReschke commented Sep 10, 2015

PVince81 commented Sep 17, 2015

blizzz commented Sep 25, 2015

4001982248998 commented Nov 27, 2015

4001982248998 commented Nov 27, 2015

PVince81 commented Mar 9, 2016

PVince81 commented Mar 9, 2016

PVince81 commented Mar 9, 2016

4001982248998 commented Mar 10, 2016

4001982248998 commented Mar 10, 2016

4001982248998 commented Mar 14, 2016

icewind1991 commented Mar 16, 2016

lock bot commented Aug 5, 2019

occ files:scan bails out with “StorageNotAvailableException” #18929

occ files:scan bails out with “StorageNotAvailableException” #18929

Comments

4001982248998 commented Sep 9, 2015

LukasReschke commented Sep 9, 2015

4001982248998 commented Sep 10, 2015

Steps to reproduce

Expected behaviour

Actual behaviour

Server configuration

LDAP configuration (delete this part if not used)

Client configuration

Logs

Web server error log

ownCloud log (data/owncloud.log)

Browser log

LukasReschke commented Sep 10, 2015

PVince81 commented Sep 17, 2015

blizzz commented Sep 25, 2015

4001982248998 commented Nov 27, 2015

4001982248998 commented Nov 27, 2015

PVince81 commented Mar 9, 2016

PVince81 commented Mar 9, 2016

PVince81 commented Mar 9, 2016

4001982248998 commented Mar 10, 2016

4001982248998 commented Mar 10, 2016

4001982248998 commented Mar 14, 2016

icewind1991 commented Mar 16, 2016

lock bot commented Aug 5, 2019