fulltextsearch:index throws unhandled exception

[budachst]

Steps to reproduce

1. install full text search
2. run sudo -u php ./occ files_fulltextsearch:index

Expected behaviour

Perform initial index of all 253736 files

Actual behaviour

Indexing starts and after some time, the process throws an unhandled exception:

An unhandled exception has been thrown:
Error: Call to a member function instanceOfStorage() on null in /var/www/nextcloud/lib/private/Files/View.php:1920
Stack trace:
#0 /var/www/nextcloud/lib/private/Files/View.php(2034): OC\Files\View->lockPath('/henning.muelle...', 1, false)
#1 /var/www/nextcloud/lib/private/Files/View.php(1134): OC\Files\View->lockFile('/henning.muelle...', 1)
#2 /var/www/nextcloud/lib/private/Files/View.php(267): OC\Files\View->basicOperation('mkdir', '/henning.muelle...', Array)
#3 /var/www/nextcloud/lib/private/Files/Node/Folder.php(160): OC\Files\View->mkdir('/henning.muelle...')
#4 /var/www/nextcloud/lib/private/Files/Node/Root.php(380): OC\Files\Node\Folder->newFolder('/henning.muelle...')
#5 [internal function]: OC\Files\Node\Root->getUserFolder('henning.mueller...')
#6 /var/www/nextcloud/lib/private/Files/Node/LazyRoot.php(64): call_user_func_array(Array, Array)
#7 /var/www/nextcloud/lib/private/Files/Node/LazyRoot.php(281): OC\Files\Node\LazyRoot->__call('getUserFolder', Array)
#8 /var/www/nextcloud/apps/files_fulltextsearch/lib/Service/FilesService.php(336): OC\Files\Node\LazyRoot->getUserFolder('henning.mueller...')
#9 /var/www/nextcloud/apps/files_fulltextsearch/lib/Service/FilesService.php(580): OCA\Files_FullTextSearch\Service\FilesService->getPathFromViewerId(3872920, 'henning.mueller...')
#10 /var/www/nextcloud/apps/files_fulltextsearch/lib/Service/FilesService.php(519): OCA\Files_FullTextSearch\Service\FilesService->updateShareNames(Object(OCA\Files_FullTextSearch\Model\FilesDocument), Object(OC\Files\Node\File))
#11 /var/www/nextcloud/apps/files_fulltextsearch/lib/Service/FilesService.php(495): OCA\Files_FullTextSearch\Service\FilesService->updateDocumentAccess(Object(OCA\Files_FullTextSearch\Model\FilesDocument), Object(OC\Files\Node\File))
#12 /var/www/nextcloud/apps/files_fulltextsearch/lib/Service/FilesService.php(474): OCA\Files_FullTextSearch\Service\FilesService->updateFilesDocumentFromFile(Object(OCA\Files_FullTextSearch\Model\FilesDocument), Object(OC\Files\Node\File))
#13 /var/www/nextcloud/apps/files_fulltextsearch/lib/Service/FilesService.php(367): OCA\Files_FullTextSearch\Service\FilesService->updateFilesDocument(Object(OCA\Files_FullTextSearch\Model\FilesDocument))
#14 /var/www/nextcloud/apps/files_fulltextsearch/lib/Provider/FilesProvider.php(239): OCA\Files_FullTextSearch\Service\FilesService->generateDocuments(Array)
#15 /var/www/nextcloud/apps/fulltextsearch/lib/Service/IndexService.php(247): OCA\Files_FullTextSearch\Provider\FilesProvider->fillIndexDocuments(Array)
#16 /var/www/nextcloud/apps/fulltextsearch/lib/Service/IndexService.php(217): OCA\FullTextSearch\Service\IndexService->indexChunk(Object(OCA\FullTextSearch_ElasticSearch\Platform\ElasticSearchPlatform), Object(OCA\Files_FullTextSearch\Provider\FilesProvider), Array)
#17 /var/www/nextcloud/apps/fulltextsearch/lib/Service/IndexService.php(126): OCA\FullTextSearch\Service\IndexService->indexChunks(Object(OCA\FullTextSearch_ElasticSearch\Platform\ElasticSearchPlatform), Object(OCA\Files_FullTextSearch\Provider\FilesProvider), Array)
#18 /var/www/nextcloud/apps/fulltextsearch/lib/Command/Index.php(148): OCA\FullTextSearch\Service\IndexService->indexProviderContentFromUser(Object(OCA\FullTextSearch_ElasticSearch\Platform\ElasticSearchPlatform), Object(OCA\Files_FullTextSearch\Provider\FilesProvider), 'alireza.davatga...')
#19 /var/www/nextcloud/apps/fulltextsearch/lib/Command/Index.php(119): OCA\FullTextSearch\Command\Index->indexProvider(Object(OCA\Files_FullTextSearch\Provider\FilesProvider))
#20 /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php(264): OCA\FullTextSearch\Command\Index->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#21 /var/www/nextcloud/core/Command/Base.php(161): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#22 /var/www/nextcloud/3rdparty/symfony/console/Application.php(874): OC\Core\Command\Base->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#23 /var/www/nextcloud/3rdparty/symfony/console/Application.php(228): Symfony\Component\Console\Application->doRunCommand(Object(OCA\FullTextSearch\Command\Index), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#24 /var/www/nextcloud/3rdparty/symfony/console/Application.php(130): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#25 /var/www/nextcloud/lib/private/Console/Application.php(173): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#26 /var/www/nextcloud/console.php(90): OC\Console\Application->run()
#27 /var/www/nextcloud/occ(11): require_once('/var/www/nextcl...')
#28 {main}

Running the indexer again will get over that one, but will throw another execption somewhere else.

Server configuration detail

Operating system: Linux 3.10.0-862.3.2.el7.x86_64 #1 SMP Mon May 21 23:36:36 UTC 2018 x86_64

Webserver: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.0.30 (apache2handler)

Database: mysql 5.5.56

PHP version: 7.0.30
Modules loaded: Core, date, libxml, openssl, pcre, zlib, filter, hash, Reflection, SPL, session, standard, apache2handler, bz2, calendar, ctype, curl, dom, mbstring, fileinfo, ftp, gd, gettext, iconv, json, ldap, exif, mysqlnd, PDO, Phar, posix, shmop, SimpleXML, sockets, sqlite3, sysvmsg, sysvsem, sysvshm, tokenizer, xml, xmlwriter, xsl, zip, mysqli, pdo_mysql, pdo_sqlite, wddx, xmlreader, apcu, igbinary, memcached, redis, Zend OPcache

Nextcloud version: 13.0.4 - 13.0.4.0

Updated from an older Nextcloud/ownCloud or fresh install:

Where did you install Nextcloud from:

Signing status

Array

List of activated apps

Enabled:
 - activity: 2.6.1
 - admin_audit: 1.3.0
 - admin_notifications: 1.0.1
 - announcementcenter: 3.2.1
 - bruteforcesettings: 1.0.3
 - calendar: 1.6.1
 - comments: 1.3.0
 - contacts: 2.1.5
 - dav: 1.4.7
 - drawio: 0.8.9
 - federatedfilesharing: 1.3.1
 - federation: 1.3.0
 - files: 1.8.0
 - files_fulltextsearch: 0.8.2
 - files_pdfviewer: 1.2.1
 - files_sharing: 1.5.0
 - files_texteditor: 2.5.1
 - files_trashbin: 1.3.0
 - files_versions: 1.6.0
 - files_videoplayer: 1.2.0
 - firstrunwizard: 2.2.1
 - fulltextsearch: 0.8.1
 - fulltextsearch_elasticsearch: 0.8.0
 - gallery: 18.0.0
 - groupfolders: 1.2.0
 - issuetemplate: 0.3.0
 - logreader: 2.0.0
 - lookup_server_connector: 1.1.0
 - nextcloud_announcements: 1.2.0
 - nextnote: 1.2.4
 - notifications: 2.1.2
 - oauth2: 1.1.1
 - onlyoffice: 1.3.0
 - passman: 2.1.4
 - password_policy: 1.3.0
 - provisioning_api: 1.3.0
 - richdocuments: 2.0.9
 - serverinfo: 1.3.0
 - sharebymail: 1.3.0
 - survey_client: 1.1.0
 - systemtags: 1.3.0
 - theming: 1.4.5
 - twofactor_backupcodes: 1.2.3
 - twofactor_totp: 1.4.1
 - twofactor_u2f: 1.5.5
 - updatenotification: 1.3.0
 - user_ldap: 1.3.1
 - workflowengine: 1.3.0
Disabled:
 - encryption
 - files_external
 - files_retention
 - user_external

Configuration (config/config.php)

{
    "instanceid": "***REMOVED SENSITIVE VALUE***",
    "passwordsalt": "***REMOVED SENSITIVE VALUE***",
    "secret": "***REMOVED SENSITIVE VALUE***",
    "trusted_domains": [
        "cloudnine.jvm.de",
        "collabora.jvm.de"
    ],
    "datadirectory": "***REMOVED SENSITIVE VALUE***",
    "overwrite.cli.url": "https:\/\/cloudnine.jvm.de",
    "overwritehost": "cloudnine.jvm.de",
    "dbtype": "mysql",
    "version": "13.0.4.0",
    "dbname": "***REMOVED SENSITIVE VALUE***",
    "dbhost": "***REMOVED SENSITIVE VALUE***",
    "dbtableprefix": "oc_",
    "dbuser": "***REMOVED SENSITIVE VALUE***",
    "dbpassword": "***REMOVED SENSITIVE VALUE***",
    "logtimezone": "UTC",
    "installed": true,
    "ldapIgnoreNamingRules": false,
    "mail_from_address": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpmode": "sendmail",
    "mail_domain": "***REMOVED SENSITIVE VALUE***",
    "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpport": "25",
    "preview_libreoffice_path": "\/opt\/libreoffice4.4\/program\/soffice",
    "enable_previews": true,
    "user_sso_auth_endpoint": "https:\/\/id.jvm.de\/_id\/auth\/authorize",
    "user_sso_token_endpoint": "http:\/\/openidconnect.jvm.de:8080\/c2id\/token",
    "user_sso_userinfo_endpoint": "http:\/\/openidconnect.jvm.de:8080\/c2id\/userinfo",
    "user_sso_client_id": "***REMOVED SENSITIVE VALUE***",
    "user_sso_client_secret": "***REMOVED SENSITIVE VALUE***",
    "user_sso_callback_uri": "https:\/\/cloudnine.jvm.de\/index.php\/apps\/user_sso\/callback",
    "loglevel": 0,
    "logfile": "\/var\/log\/nextcloud.log",
    "theme": "",
    "memcache.locking": "\\OC\\Memcache\\Redis",
    "memcache.local": "\\OC\\Memcache\\APCu",
    "memcache.distributed": "\\OC\\Memcache\\Memcached",
    "memcached_servers": [
        [
            "localhost",
            11211
        ]
    ],
    "redis": {
        "host": "***REMOVED SENSITIVE VALUE***",
        "port": "6379",
        "timeout": "1.5",
        "dbindex": 4
    },
    "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
    "forwarded_for_headers": [
        "HTTP_X_FORWARDED_FOR",
        "HTTP_FORWARDED_FOR"
    ],
    "trashbin_retention_obligation": "7,30",
    "maintenance": false,
    "ldapProviderFactory": "\\OCA\\User_LDAP\\LDAPProviderFactory",
    "updater.release.channel": "production",
    "mail_smtpauthtype": "LOGIN"
}

Are you using external storage, if yes which one: local/smb/sftp/...

Are you using encryption: no

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...

LDAP configuration (delete this part if not used)

With access to your command line run e.g.:
sudo -u www-data php occ ldap:show-config
from within your Nextcloud installation folder

Without access to your command line download the data/owncloud.db to your local
computer or access your SQL server remotely and run the select query:
SELECT * FROM `oc_appconfig` WHERE `appid` = 'user_ldap';


Eventually replace sensitive data as the name/IP-address of your LDAP server or groups.

Client configuration

Browser: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:62.0) Gecko/20100101 Firefox/62.0

Operating system:

Logs

Browser log

Insert your webserver log here 

Nextcloud log

Insert your Nextcloud log here

Browser log

Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log
c) ...

[budachst]

This error seems to stem from some weird LDAP issues, when trying to resolve shared files. I noticed that each time, this issue occurs, there is a corresponding error like this in the nextcloud log:

{"reqId":"IhUiXP9Cx3KZy2anhndK","level":0,"time":"2018-06-26T12:26:10+00:00","remoteAddr":"","user":"--","app":"user_ldap","method":"--","url":"--","message":"readAttribute failed for DN uid=<username>,<domain>","userAgent":"--","version":"13.0.4.0"}

And in fact, the indexer just died like this:

An unhandled exception has been thrown:
Error: Call to a member function instanceOfStorage() on null in /var/www/nextcloud/lib/private/Files/View.php:1920
Stack trace:
#0 /var/www/nextcloud/lib/private/Files/View.php(2034): OC\Files\View->lockPath('/<username>...', 1, false)
#1 /var/www/nextcloud/lib/private/Files/View.php(1134): OC\Files\View->lockFile('/<username>...', 1)
#2 /var/www/nextcloud/lib/private/Files/View.php(267): OC\Files\View->basicOperation('mkdir', '/<username>...', Array)
#3 /var/www/nextcloud/lib/private/Files/Node/Folder.php(160): OC\Files\View->mkdir('/<username>...')
#4 /var/www/nextcloud/lib/private/Files/Node/Root.php(380): OC\Files\Node\Folder->newFolder('/<username>...')
#5 [internal function]: OC\Files\Node\Root->getUserFolder('<username>')
#6 /var/www/nextcloud/lib/private/Files/Node/LazyRoot.php(64): call_user_func_array(Array, Array)
#7 /var/www/nextcloud/lib/private/Files/Node/LazyRoot.php(281): OC\Files\Node\LazyRoot->__call('getUserFolder', Array)
#8 /var/www/nextcloud/apps/files_fulltextsearch/lib/Service/FilesService.php(336): OC\Files\Node\LazyRoot->getUserFolder('<username>')
#9 /var/www/nextcloud/apps/files_fulltextsearch/lib/Service/FilesService.php(580): OCA\Files_FullTextSearch\Service\FilesService->getPathFromViewerId(1206790, '<username>')

[budachst]

I have tried to circumvent this issue by providing multiple LDAP replicas, but I can't get rid of this issue. Could the indexer made to wait a bit for the LDAP server to answer? Or maybe only perform a soft error and skip the current account, but continue indexing the remaining accounts, because the way it works now, I won't be able to finish one initial index ever.

[truelai]

I had the same issue. Coming to this post actually helped me solve it as I didn't think of the relationship between LDAP and the external mounts I was indexing (mounted using LDAP).

when I checked my LDAP settings, I realized that things seemed a bit janky. Auth was completely inconsistent. I then noticed that the release notes for NC 13.0.05 had addressed some LDAP issues, so I removed my LDAP server, upgraded to 13.0.05, and added the LDAP server again.

Logs still show inconsistency in ability to auth, but after resetting my index, I am now flying like I've never seen and am not getting hung up on the users with access to the external mounts. So, my advice is, if you see RAM just repeating and not increasing in value, have external mounts, and use LDAP ... upgrade to NC13.05, remove and re-add your LDAP server, and reset and re-index.

I'm still in the process of indexing (> 100K docs) but I'll update this when it completes. Its looks good thus far, though.

[budachst]

Thanks - I am already on NC13.0.5, but resetting LDAP surely couldn't hurt. I will try that out and see, how it goes…

[truelai]

@budachst - I also installed Kibana so I can have further insight into what's happening and when. My index completed and replicated across the cluster. This is definitely a first for me. LDAP seems to have really been the culprit.

Even now, when configuring LDAP, I get inconsistent auth but 13.0.05 seems to handle those failures better than previously and doesn't get stuck on external drives mounted with LDAP.

[budachst]

I now got fulltextsearch churning away on my test instance and it indexed approx. 1/3 of my accounts (400/1200) so far without issue. I have done two things: remove the uid attribute from the user home folder naming section, which repeatedly causes an ldap attribut lookup error and changed the ldap configuration to use another of our ldap servers. I probably will also setup an caching ldap server on our NC instance itself to get even more load off our backend ldap servers.

[ArtificialOwl]

Any update on your tests ? Can we close this ticket ?

[budachst]

Yes - this ticket can be closed.