Files added to other users shared folder does not count immediately

[luizluca]

Hello

I'm having problems with OC8.0 regard the quota with shared folders. Files added to other user's shared folder does not count until I manually rescan the folder.

What happens:

1. create user A with 100MB
2. create user B with 1000MB
3. user A creates folder F
4. user A shares folder F with user B
5. user A adds 10MB of files to folder F (its quota uses 10MB)
6. user B adds 50MB of files to folder F (not its quota nor user A quota is used)
   6a. Acoording to past OC doc, A should have its quota used (https://doc.owncloud.org/server/6.0/user_manual/files/quota.html), and this is what I wanted
   6b. Acoording to OC8.0 doc, B should have its quota used (https://doc.owncloud.org/server/8.0/user_manual/files/quota.html)
7. user B adds 50MB of files to folder F (not its quota nor user A quota is used)

The expected result would be:
6a. user B adds 50MB of files to folder F (user A quota is used)
or
6b. user B adds 50MB of files to folder F (user B quota is used)

In case of 6a:
7. user B adds 50MB of files to folder F. Fails with over quota usage

If I trigger a file-scan manually for user A (./occ files:scan --path="/A/"), its quota is updated correctly according to 6a (not 6b) (It is not what docs say but I guess it makes a little more sense).
If 7. is retried after the rescan, it recieves an over quota error.

The same problem of not updating user A quota happens when user B removes a file from folder F.

It seems that, at least, the quota checking is correct. At step 6., if user try to upload 100MB, OC avoids telling that there is not enough space. However, I can upload less than 90MB multiple times as user A quota is not updated.

[PVince81]

Do you have a very deep folder structure ?

Normally upon file upload, the size is updated for every parent folder up to the "files" folder in the database (you can check it in oc_filecache where path = 'files').

Maybe in your case something is delaying that update.
Did you find anything in owncloud.log, potential errors that might have aborted the size propagation ?

[nickvergessen]

Can you also post your error log and owncloud log? Maybe they contain related stuff

[nickvergessen]

@PVince81 related to #14516 ? Could be the same issue, no?

[luizluca]

error.log:

[Mon Mar 02 17:44:51.618520 2015] [authz_core:error] [pid 26827] [client 10.9.7.157:38121] AH01630: client denied by server configuration: /srv/www/htdocs/owncloud/data/htaccesstest.txt
[Mon Mar 02 17:48:16.543558 2015] [authz_core:error] [pid 4192] [client 10.9.7.157:38482] AH01630: client denied by server configuration: /srv/www/htdocs/owncloud/data/htaccesstest.txt
(repeated many times)

owncloud.log (during the upload of two files):

{"reqId":"54822935002afb795129968c0c2823fd","remoteAddr":"10.9.200.2","app":"lock","message":"INFO: Write lock requested for \/srv\/www\/htdocs\/owncloud\/data\/_csit-scd\/files\/csit-scd\/Faixa 05.flac","level":0,"time":"2015-03-02T21:03:37+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"54822935002afb795129968c0c2823fd","remoteAddr":"10.9.200.2","app":"lock","message":"PASS: Write lock succeeded on \/srv\/www\/htdocs\/owncloud\/data\/_csit-scd\/files\/csit-scd\/Faixa 05.flac","level":0,"time":"2015-03-02T21:03:37+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"54822935002afb795129968c0c2823fd","remoteAddr":"10.9.200.2","app":"lock","message":"INFO: Incrementing lock type 2 count for \/srv\/www\/htdocs\/owncloud\/data\/_csit-scd\/files\/csit-scd\/Faixa 05.flac","level":0,"time":"2015-03-02T21:03:37+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"54822935002afb795129968c0c2823fd","remoteAddr":"10.9.200.2","app":"files_sharing","message":"can not update etags on \/files\/ for user eron. Path does not exists","level":0,"time":"2015-03-02T21:03:38+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"54822935002afb795129968c0c2823fd","remoteAddr":"10.9.200.2","app":"files_sharing","message":"can not update etags on \/files\/ for user joaoal. Path does not exists","level":0,"time":"2015-03-02T21:03:38+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"54822935002afb795129968c0c2823fd","remoteAddr":"10.9.200.2","app":"files_sharing","message":"can not update etags on \/files\/ for user mossmann. Path does not exists","level":0,"time":"2015-03-02T21:03:38+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"54822935002afb795129968c0c2823fd","remoteAddr":"10.9.200.2","app":"files_sharing","message":"can not update etags on \/files\/ for user reyel. Path does not exists","level":0,"time":"2015-03-02T21:03:38+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"54822935002afb795129968c0c2823fd","remoteAddr":"10.9.200.2","app":"files_sharing","message":"can not update etags on \/files\/ for user slmoser. Path does not exists","level":0,"time":"2015-03-02T21:03:38+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"54822935002afb795129968c0c2823fd","remoteAddr":"10.9.200.2","app":"files_sharing","message":"can not update etags on \/files\/ for user eron. Path does not exists","level":0,"time":"2015-03-02T21:03:38+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"54822935002afb795129968c0c2823fd","remoteAddr":"10.9.200.2","app":"files_sharing","message":"can not update etags on \/files\/ for user joaoal. Path does not exists","level":0,"time":"2015-03-02T21:03:38+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"54822935002afb795129968c0c2823fd","remoteAddr":"10.9.200.2","app":"files_sharing","message":"can not update etags on \/files\/ for user mossmann. Path does not exists","level":0,"time":"2015-03-02T21:03:38+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"54822935002afb795129968c0c2823fd","remoteAddr":"10.9.200.2","app":"files_sharing","message":"can not update etags on \/files\/ for user reyel. Path does not exists","level":0,"time":"2015-03-02T21:03:38+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"54822935002afb795129968c0c2823fd","remoteAddr":"10.9.200.2","app":"files_sharing","message":"can not update etags on \/files\/ for user slmoser. Path does not exists","level":0,"time":"2015-03-02T21:03:38+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c6811e3659708cb89bbac0a456a365","remoteAddr":"10.9.200.2","app":"lock","message":"INFO: Write lock requested for \/srv\/www\/htdocs\/owncloud\/data\/_csit-scd\/files\/csit-scd\/Faixa 06.flac","level":0,"time":"2015-03-02T21:03:38+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c6811e3659708cb89bbac0a456a365","remoteAddr":"10.9.200.2","app":"lock","message":"PASS: Write lock succeeded on \/srv\/www\/htdocs\/owncloud\/data\/_csit-scd\/files\/csit-scd\/Faixa 06.flac","level":0,"time":"2015-03-02T21:03:38+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c6811e3659708cb89bbac0a456a365","remoteAddr":"10.9.200.2","app":"lock","message":"INFO: Incrementing lock type 2 count for \/srv\/www\/htdocs\/owncloud\/data\/_csit-scd\/files\/csit-scd\/Faixa 06.flac","level":0,"time":"2015-03-02T21:03:38+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c6811e3659708cb89bbac0a456a365","remoteAddr":"10.9.200.2","app":"files_sharing","message":"can not update etags on \/files\/ for user eron. Path does not exists","level":0,"time":"2015-03-02T21:03:39+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c6811e3659708cb89bbac0a456a365","remoteAddr":"10.9.200.2","app":"files_sharing","message":"can not update etags on \/files\/ for user joaoal. Path does not exists","level":0,"time":"2015-03-02T21:03:39+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c6811e3659708cb89bbac0a456a365","remoteAddr":"10.9.200.2","app":"files_sharing","message":"can not update etags on \/files\/ for user mossmann. Path does not exists","level":0,"time":"2015-03-02T21:03:39+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c6811e3659708cb89bbac0a456a365","remoteAddr":"10.9.200.2","app":"files_sharing","message":"can not update etags on \/files\/ for user reyel. Path does not exists","level":0,"time":"2015-03-02T21:03:39+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c6811e3659708cb89bbac0a456a365","remoteAddr":"10.9.200.2","app":"files_sharing","message":"can not update etags on \/files\/ for user slmoser. Path does not exists","level":0,"time":"2015-03-02T21:03:39+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c6811e3659708cb89bbac0a456a365","remoteAddr":"10.9.200.2","app":"files_sharing","message":"can not update etags on \/files\/ for user eron. Path does not exists","level":0,"time":"2015-03-02T21:03:39+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c6811e3659708cb89bbac0a456a365","remoteAddr":"10.9.200.2","app":"files_sharing","message":"can not update etags on \/files\/ for user joaoal. Path does not exists","level":0,"time":"2015-03-02T21:03:39+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c6811e3659708cb89bbac0a456a365","remoteAddr":"10.9.200.2","app":"files_sharing","message":"can not update etags on \/files\/ for user mossmann. Path does not exists","level":0,"time":"2015-03-02T21:03:39+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c6811e3659708cb89bbac0a456a365","remoteAddr":"10.9.200.2","app":"files_sharing","message":"can not update etags on \/files\/ for user reyel. Path does not exists","level":0,"time":"2015-03-02T21:03:39+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}
{"reqId":"53c6811e3659708cb89bbac0a456a365","remoteAddr":"10.9.200.2","app":"files_sharing","message":"can not update etags on \/files\/ for user slmoser. Path does not exists","level":0,"time":"2015-03-02T21:03:39+00:00","method":"POST","url":"\/owncloud\/index.php\/apps\/files\/ajax\/upload.php"}

All users mentioned in "can not update etags on /files/ for user xxxx" are users in LDAP who did not already logged into owncloud.

[luizluca]

BTW, I can trigger the "quota update" adding a new folder (anywhere) as the user that shares the directory.

[luizluca]

If you point me where in OC code it should update the user quota info, I can debug it myself.

[PVince81]

Hmmm that is a strange set of errors...

The code that updates the "size" for all files, which is used for the used space calculation, is here: https://github.com/owncloud/core/blob/master/lib/private/files/cache/cache.php#L597

[luizluca]

Got a debugger running :-)

It seems that oc_filecache is updated as expected. The problem is only with the sharer quota calculation. Where is the user quota updated?

Initial:
 fileid | storage |      path      |            path_hash             | parent |   name   | mimetype | mimepart | size |   mtime    | storage_mtime | encrypted | unencrypted_size |     etag      | permissions 
--------+---------+----------------+----------------------------------+--------+----------+----------+----------+------+------------+---------------+-----------+------------------+---------------+-------------
    701 |      49 | files/csit-scd | 392de4c929ac439efbd31ebce5cd4af0 |    693 | csit-scd |        2 |        1 |    0 | 1425418901 |    1425418901 |         0 |                0 | 54f62a959359b |          31

user1_quota = 2.4M

event: user2 add /files/csit-scd/Faixa 01.flac (22.837.674 bytes)

 fileid | storage |             path             |            path_hash             | parent |     name      | mimetype | mimepart | size |   mtime    | storage_mtime | encrypted | unencrypted_size |               etag               | permissions 
--------+---------+------------------------------+----------------------------------+--------+---------------+----------+----------+------+------------+---------------+-----------+------------------+----------------------------------+-------------
    912 |      49 | files/csit-scd/Faixa 01.flac | 52a23a49935317fe5400c83e670461c7 |    701 | Faixa 01.flac |       22 |       16 |    0 | 1425419844 |    1425419844 |         0 |                0 | fd4dccaa63bdeb32374c124fe2efaec1 |          27
    701 |      49 | files/csit-scd               | 392de4c929ac439efbd31ebce5cd4af0 |    693 | csit-scd      |        2 |        1 |    0 | 1425419844 |    1425419844 |         0 |                0 | 54f62e33e8e28    

# from correctFolderSize, first ref
(1st call) calculateFolderSize ($path = files/csit-scd/Faixa 01.flac, $entry = null)
 - nothing happens as it is not a folder

# from correctFolderSize, second ref
(2st call) calculateFolderSize ($path = files/csit-scd, $entry = ...)
- nothing update as file size is still 0 in DB

(3rd call) calculateFolderSize ($path = files/csit-scd/Faixa 01.flac, $entry = null)
- nothing happens as it is not a folder

 fileid | storage |             path             |            path_hash             | parent |     name      | mimetype | mimepart |   size   |   mtime    | storage_mtime | encrypted | unencrypted_size |               etag               | permissions 
--------+---------+------------------------------+----------------------------------+--------+---------------+----------+----------+----------+------------+---------------+-----------+------------------+----------------------------------+-------------
    912 |      49 | files/csit-scd/Faixa 01.flac | 52a23a49935317fe5400c83e670461c7 |    701 | Faixa 01.flac |       22 |       16 | 22837674 | 1425419902 |    1425419902 |         0 |                0 | 93b62906a770f19a2c2ea465003bbaa0 |          27
    701 |      49 | files/csit-scd               | 392de4c929ac439efbd31ebce5cd4af0 |    693 | csit-scd      |        2 |        1 |        0 | 1425419844 |    1425419844 |         0 |                0 | 54f62e7e46ca4                    |          31
(2 registros)


(4th call) calculateFolderSize ($path = files/csit-scd, $entry = ...)
  called $this->updated(912,22837674)

 fileid | storage |             path             |            path_hash             | parent |     name      | mimetype | mimepart |   size   |   mtime    | storage_mtime | encrypted | unencrypted_size |               etag               | permissions 
--------+---------+------------------------------+----------------------------------+--------+---------------+----------+----------+----------+------------+---------------+-----------+------------------+----------------------------------+-------------
    912 |      49 | files/csit-scd/Faixa 01.flac | 52a23a49935317fe5400c83e670461c7 |    701 | Faixa 01.flac |       22 |       16 | 22837674 | 1425419902 |    1425419902 |         0 |                0 | 93b62906a770f19a2c2ea465003bbaa0 |          27
    701 |      49 | files/csit-scd               | 392de4c929ac439efbd31ebce5cd4af0 |    693 | csit-scd      |        2 |        1 | 22837674 | 1425420044 |    1425419844 |         0 |                0 | 54f62f0c07195      

user1_quota = 2.4M (still)

[PVince81]

@luizluca the usage space of "user1" is calculated by looking at the size of the "files" entry (path = 'files') in the database. Then take the user's quota from the users page that was set by the admin and substract. So "free_space = quota_from_the_users_page - usage_space".

[luizluca]

@PVince81 , I though that files might hold the user used space after I was going home... :-)

So, I guess I discovered the problem and I'm starting to ask if ever it worked in any owncloud setup...

The folder size is updated using this function:

OC\Files\Cache\Cache->correctFolderSize() at /srv/www/htdocs/owncloud/lib/private/files/cache/cache.php:569
    public function correctFolderSize($path, $data = null) {
        $this->calculateFolderSize($path, $data);
        if ($path !== '') {
            $parent = dirname($path);
            if ($parent === '.' or $parent === '/') {
                $parent = '';
            }
            $this->correctFolderSize($parent);
        }
    }

Where calculateFolderSize can be implemented by Cache or its decedents: HomeCache and Shared_Cache (which calls cache::correctFolderSize). Their implementation are correct, as folder size is updated when called.

correctFolderSize updates size for a path and recursively calls itself until path is ''. For a user accessing its own folder, I get these calls:

user1:
correctFolderSize('files/csit-scd/Faixa 01.flac', $data)
correctFolderSize('files/csit-scd', $data)
correctFolderSize('files', $data)
correctFolderSize('', $data)

info: actually, it is all those twice. The first with filesize=0 (file creation?) and the second one with
filesize correct.

While when another user writes to the shared csit-scd folder, I get:

user2:
correctFolderSize('Faixa 01.flac', $data)
correctFolderSize('', $data)

It never goes down the shared folder. So, calculateFolderSize is never called for user1 "files".

I do not see another place in code that does this recursion. and subclasses do not replace correctFolderSize. So, I guess there is no place for files to get updated when user2 is writing.
Maybe there is some kind of hook outside code waiting for a shared folder to be updated and replay the correctFolderSize as user1. However, if it should do, it was never called and neither hit my breakpoint.

[PVince81]

Hmm... this used to work in OC 6.

If user2 changes a file that belongs to user1, then the shared storage should propagate the size change to user1's storage.

[PVince81]

Also, to make sure there is no misunderstanding: the used space only counts for file owners. So if user1 is the owner of the "csit-csd" folder, then any file modified inside still counts in user1's quota, even if it's user2 accessing it.

[luizluca]

Yes, I guess there is no misunderstanding. the problem is that user1 quota is not changed when user2 writes to user1 shared folder.

It is only a propagation problem as any modification in any user1 files by user1 generates the updates until the files folder. At this point, any modification that user2 did inside the shared folder starts to count.