Files locks in database after occ upgrade/server restart

[mrow4a]

I have observed, that if I am stressing the ownCloud server with performance test (smashbox, automated syncing with different distribution of the files), and during that synchronisation I will start upgrade to the newest ownCloud enterprise version, after the upgrade there are hanging files in the oc_file_locks table. After the upgrade, the files have 1h lock timeout and I cannot do anything with them, I cannot also delete the parent folder.

Steps to reproduce

1. Install and configure from tar the most recent ownCloud Enterprise stable 9.1 installation
2. Run synchronisation of files against that installation
3. Upgrade to e.g. newest ownCloud Enterprise 9.1 daily master
4. Immediately start new synchronistion of the same files/adding new ones/modifying/deleting parent folder.

Expected behaviour

After the upgrade/restart there should be no problem in synchronisation against the previously locked files

Actual behaviour

The files and parent folder is locked.

Server configuration

Operating system:
Ubuntu 14.04

Database:
mysql

PHP version:
PHP 5.5.9-1ubuntu4.17

Updated from an older ownCloud or fresh install:
yes

Where did you install ownCloud from:
tar

Client configuration

Operating system:
Ubuntu 14.04

Logs

Web server error log

MariaDB [oc]> select * from oc_file_locks;
+-------+------+----------------------------------------+------------+
| id | lock | key | ttl |
+-------+------+----------------------------------------+------------+
| 31667 | 0 | files/468c740695b998c9b920835b3af18225 | 1468232951 |
| 31668 | 0 | files/b6ef755cfc70b5eb94f5d3aad8ba833d | 1468232951 |
| 33171 | 1 | files/ddee88e3731ca7baa207ab840f15c8dd | 1468256952 |
| 33172 | 1 | files/4a7d2d1427b76553905194eebf73278a | 1468256952 |
| 33173 | -1 | files/1bed30ff02ff9684ef5d6e29a41b8c91 | 1468256942 |
| 33174 | 0 | files/8979e76236531b62718378f016c99616 | 1468256954 |
+-------+------+----------------------------------------+------------+

ownCloud log (data/owncloud.log)

{"reqId":"xxx","remoteAddr":"xxx","app":"webdav","message":"Exception: {"Message":"HTTP\/1.1 423 \"test\" is locked","Exception":"OCA\DAV\Connector\Sabre\Exception\FileLocked","Code":0,"Trace":"#0 \/var\/www\/owncloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Tree.php(179): OCA\DAV\Connector\Sabre\Directory->delete()\n#1 \/var\/www\/owncloud\/3rdparty\/sabre\/dav\/lib\/DAV\/CorePlugin.php(285): Sabre\DAV\Tree->delete('test')\n#2 [internal function]: Sabre\DAV\CorePlugin->httpDelete(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))\n#3 \/var\/www\/owncloud\/3rdparty\/sabre\/event\/lib\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\n#4 \/var\/www\/owncloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(459): Sabre\Event\EventEmitter->emit('method:DELETE', Array)\n#5 \/var\/www\/owncloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(248): Sabre\DAV\Server->invokeMethod(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))\n#6 \/var\/www\/owncloud\/apps\/dav\/appinfo\/v1\/webdav.php(56): Sabre\DAV\Server->exec()\n#7 \/var\/www\/owncloud\/remote.php(164): require_once('\/var\/www\/ownclo...')\n#8 {main}","File":"\/var\/www\/owncloud\/apps\/dav\/lib\/Connector\/Sabre\/Directory.php","Line":283,"User":"xxx"}","level":4,"time":"2016-07-10T13:45:42+00:00","method":"DELETE","url":"/owncloud/remote.php/webdav/test/","user":"xxx"}

[jnweiger]

Two ideas:

• occ status should list running php background jobs. This could help with avoiding stale locks and is a first building block for further introspection and a safer upgrade procedure.
• occ maintenance:repair should check for stale locks? This could help after the issue happened. Also a building block for hardening the upgrade procedure.

@felixboehm fyi.

[mrow4a]

Suggestion:
what about implementing in occ upgrade the blocking mechanism so that it won't update if there are some apache2 php scripts being executed.

If there are some php being interupted, thus files locks are hanging in the database, what about cleaning these locks during the ownCloud startup (service apache2 start). These also requires rethinking for clustered setups.

[mrow4a]

Or "occ restart" command instead of manual apache2 restart?

[mrow4a]

It is only valid for some files which have been in some process on the server. (lock -1, 1)

[PVince81]

blocking mechanism so that it won't update if there are some apache2 php scripts being executed.

Not sure if this is even possible without a lot of hackery.

[PVince81]

There is also a slight chance that this has nothing to do with pending PHP requests. There have been reports of stray locks in some configurations but so far it wasn't possible to find out why these weren't freed.

[mrow4a]

What I did it was syncing a lot of small files and just run service apache2 restart in the middle. It resulted in some files having lock flag -1 and 1 after restart.

I solved the problem by doing apachectl -k graceful-stop with GracefulShutdownTimeout timeout.
If exceeded timeout, script continues with:

cd /var/www/owncloud/ && sudo -u www-data php occ maintenance:mode --on
mysql --user="$user" --password="$password" --execute="use $database; update oc_jobs SET last_run=0 where class='OCA\\Files\\BackgroundJob\\CleanupFileLocks'; delete from oc_file_locks;"
cd /var/www/owncloud/ && sudo -u www-data php cron.php
cd /var/www/owncloud/ && sudo -u www-data php occ maintenance:mode --off

This could be the way ownCloud is being restarted, or at least started.

[PVince81]

Still an issue ? I suggest closing unless it happens again

[ownclouders]

Hey, this issue has been closed because the label status/STALE is set and there were no updates for 7 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.