Scanner doesn't propagate etags any more when file changed #24255

Closed
PVince81 opened this Issue Apr 25, 2016 · 13 comments

Projects

None yet

4 participants

@PVince81
Collaborator

Steps:

  1. Pick an external storage type (SFTP or SMB)
  2. Directly on the storage, create a folder "updatetest/sub/test.txt"
  3. Setup that external storage in ownCloud
  4. Close the web browser to avoid interferences (buildstoragestats, etc)
  5. Run occ files:scan --all
  6. Check database: select fileid,path,size,mtime,storage_mtime,etag from oc_filecache where storage=3 order by path;. Take note of etags.
  7. Change the contents of the file "test.txt" directly on the storage, not through ownCloud.
  8. Run occ files:scan --all
  9. Check database: select fileid,path,size,mtime,storage_mtime,etag from oc_filecache where storage=3 order by path;. Compare etags.

Expected result

After the second scan, the etag and size of all parents were updated up to the storage root.

Actual result

After the second scan, only the size changed but not the etag.
Only the etag of "test.txt" changed, and so does its "storage_mtime".

Versions

Works in ownCloud 8.2.3
Broken in ownCloud 9.0.1

@icewind1991 please help fixing this regression, thanks.

CC @MorrisJobke @bboule

@PVince81 PVince81 added this to the 9.0.2-current-maintenance milestone Apr 25, 2016
@PVince81
Collaborator

This is critical because it's the only way to properly make ownCloud detect remote changes for now.

@icewind1991
Member

can reproduce

@icewind1991
Member

fix is here #24256

@PVince81
Collaborator

@icewind1991 thanks a lot. Any reason why it wasn't triggered any more ? Or did it happen indirectly through the recent changes to propagation code ?

@ghost
ghost commented Apr 25, 2016

@PVince81 Could it be possible that this is also fixing #23019 ?

@PVince81
Collaborator

@RealRancor unlikely.

@ghost
ghost commented Apr 25, 2016

Mhh, but the user there has issues with oC9 that files changed on external storages are not synced / detected by the sync client. And shouldn't this be caused if etags are not updated / correclty propagated?

@PVince81
Collaborator
PVince81 commented Apr 25, 2016 edited

Ah, that part yes. I only saw "out of memory" issues, which won't be solved by this PR.

@Kaelber
Kaelber commented May 4, 2016

Update :

with oc 9.0.2 RC-2 and oc_9.1.0 pAlp-1 and Client 2.2.0 Beta 2 the same situation :

Synology DS414 mit DSM 5.2 | ? : rescan runs and logs are storing, but no file update is running from server to client.
Synology DS712+ mit DSM 6 | ? : rescan runs and logs are storing, but no file update is running from server to client.

Regarding the filescan I got from the Logfile as follow :

With oc 9.0.1 (not 9.1.0)

Starting scan for user 1 out of 1 (rescan)

+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 12706 | 52211 | 01:04:55 |
+---------+-------+--------------+

----- End ----------

with oc 9.0.2 RC-2

Starting scan for user 1 out of 1 (rescan)

----- End ----------

@PVince81
Collaborator
PVince81 commented May 4, 2016

@Kaelber can you check the etags like I did in my original steps ?

@Kaelber
Kaelber commented May 9, 2016 edited

To test I got the problem with oc 9.0.2 from the filescan log as follows :

An unhandled exception has been thrown:
exception 'Doctrine\DBAL\DBALException' with message 'Failed to connect to the database: An exception occured in driver: could not find driver' in /volume1/web/owncloud/lib/private/db/connection.php:54
Stack trace:

#0 /volume1/web/owncloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(429): OC\DB\Connection->connect()
#1 /volume1/web/owncloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(389): Doctrine\DBAL\Connection->getDatabasePlatformVersion()
#2 /volume1/web/owncloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(328): Doctrine\DBAL\Connection->detectDatabasePlatform()
#3 /volume1/web/owncloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(621): Doctrine\DBAL\Connection->getDatabasePlatform()
#4 /volume1/web/owncloud/lib/private/db/connection.php(137): Doctrine\DBAL\Connection->setTransactionIsolation(2)
#5 /volume1/web/owncloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DriverManager.php(172): OC\DB\Connection->__construct(Array, Object(Doctrine\DBAL\Driver\PDOMySql\Driver), Object(Doctrine\DBAL\Configuration), Object(Doctrine\Common\EventManager))
#6 /volume1/web/owncloud/lib/private/db/connectionfactory.php(118): Doctrine\DBAL\DriverManager::getConnection(Array, Object(Doctrine\DBAL\Configuration), Object(Doctrine\Common\EventManager))
#7 /volume1/web/owncloud/lib/private/server.php(366): OC\DB\ConnectionFactory->getConnection('mysql', Array)
#8 /volume1/web/owncloud/3rdparty/pimple/pimple/src/Pimple/Container.php(113): OC\Server->OC\{closure}(Object(OC\Server))
#9 /volume1/web/owncloud/lib/private/appframework/utility/simplecontainer.php(102): Pimple\Container->offsetGet('DatabaseConnect...')
#10 /volume1/web/owncloud/lib/private/servercontainer.php(87): OC\AppFramework\Utility\SimpleContainer->query('DatabaseConnect...')
#11 /volume1/web/owncloud/lib/private/server.php(886): OC\ServerContainer->query('DatabaseConnect...')
#12 /volume1/web/owncloud/lib/private/server.php(260): OC\Server->getDatabaseConnection()
#13 /volume1/web/owncloud/3rdparty/pimple/pimple/src/Pimple/Container.php(113): OC\Server->OC\{closure}(Object(OC\Server))
#14 /volume1/web/owncloud/lib/private/appframework/utility/simplecontainer.php(102): Pimple\Container->offsetGet('AppConfig')
#15 /volume1/web/owncloud/lib/private/servercontainer.php(87): OC\AppFramework\Utility\SimpleContainer->query('AppConfig')
#16 /volume1/web/owncloud/lib/private/server.php(825): OC\ServerContainer->query('AppConfig')
#17 /volume1/web/owncloud/lib/private/server.php(411): OC\Server->getAppConfig()
#18 /volume1/web/owncloud/3rdparty/pimple/pimple/src/Pimple/Container.php(113): OC\Server->OC\{closure}(Object(OC\Server))
#19 /volume1/web/owncloud/lib/private/appframework/utility/simplecontainer.php(102): Pimple\Container->offsetGet('AppManager')
#20 /volume1/web/owncloud/lib/private/servercontainer.php(87): OC\AppFramework\Utility\SimpleContainer->query('AppManager')
#21 /volume1/web/owncloud/lib/private/server.php(1063): OC\ServerContainer->query('AppManager')
#22 /volume1/web/owncloud/lib/private/app.php(254): OC\Server->getAppManager()
#23 /volume1/web/owncloud/lib/private/app.php(103): OC_App::getEnabledApps()
#24 /volume1/web/owncloud/lib/base.php(577): OC_App::loadApps(Array)
#25 /volume1/web/owncloud/lib/base.php(1112): OC::init()
#26 /volume1/web/owncloud/console.php(46): require_once('/volume1/web/ow...')
#27 /volume1/web/owncloud/occ(11): require_once('/volume1/web/ow...')
#28 {main}
@PVince81
Collaborator
PVince81 commented May 9, 2016

@Kaelber it looks like you're having environment problems with your database

@Kaelber
Kaelber commented May 30, 2016

Update :

with oc 9.1.0 Beta 1 and Client 2.2.0 the follow situation :

Synology DS414 mit DSM 5.2 | ? : scan runs and logs are storing, file only updating, if I use --all in the scan command. A USER in the command is NOT RUNNING, no update is done.

Synology DS712+ mit DSM 6 | ? : rescan runs and logs are storing, but no file update is running from server to client, also not with --all in the scan command

Regarding the filescan I got from the Logfile as follow :

Synology DS414 with DSM 5.2

Scanning files for 4 users
Starting scan for user 1 out of 4 (admin)
Starting scan for user 2 out of 4 (kaelber)
Starting scan for user 3 out of 4 (rescan)
Starting scan for user 4 out of 4 (kaelber2)

+---------+--------+--------------+
| Folders | Files | Elapsed time |
+---------+--------+--------------+
| 54235 | 238428 | 03:05:59 |
+---------+--------+--------------+

----- End ----------

Synology DS712+ with DSM 6

Scanning files for 4 users
Starting scan for user 1 out of 4 (admin)
Starting scan for user 2 out of 4 (kaelber)

----- End ----------

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment