Sync Protocol: use oc:size instead of d:quota-used-bytes when browsing folders

[moscicki]

With 2.1 client when the folders are browsed (selective sync) the underlying PROPFIND queries for d:quota-used-bytes (ref: https://github.com/cernbox/smashbox/blob/master/protocol/protocol.md#quota-check-call)

It should not do so because:

1. another property is already clearly defined for this purpose: oc:size
2. the meaning of d:quota-used-bytes is not generally well defined for a directory when files of different file owners are in the tree

So for a storage system d:quota-used-bytes this is a property of the account (or quota node) but not that of the individual directory.

In the future I would consider to only use d:quota-used-bytes if you really want to know how much of his quota the user is using (as opposed to checking how big is the directory tree -- aka recursive accounting).

This is not a show-stopper for us currently because we can change the meaning of d:quota-used-bytes on the server but long-term this is not a good idea.

[guruz]

I remember @icewind1991 @PVince81 @ogoffart discussing about this..

[ogoffart]

I did not know about oc:size. It's clearly what we should be using i guess.

[PVince81]

From what I see in core, "oc:size" calls getSize() on the Directory Sabre Node, which would return the value of the "size" column in oc_filecache. See https://github.com/owncloud/core/blob/v8.2.2/lib/private/connector/sabre/node.php#L190

The "quota-used-bytes" will retrieve the storage's used bytes, regardless which path you are querying.
However, there might be a difference if the path in question is on another external storage, then you'd get the used bytes on that specific storage. See https://github.com/owncloud/core/blob/v8.2.2/lib/private/connector/sabre/directory.php#L265

[PVince81]

One more thing: the "size" column in oc_filecache is the sum of the size of all contents inside the queried folder, and this for the current storage. If there are mount points inside that folder (received shares, external storages), they are not counted in the size.

[ogoffart]

Yes, we want the size. As in "How much harddrive space would it take if we sync this folder"

[moscicki]

I propose to first settle on the protocol and then map it to the server implementation. My proposal is that
oc:size is used to discover the space occupied by the folder (a maximum space required to synchronize the data [maximum = taking into account that maybe some parts of the tree may not be accessed due to permissions]) and I would leave it up to the server internally to handle the case of remotely mounted storage (via whatever internal method it likes, caching etc).

Alternative is to make it non-transparent to the client such that the client is capable of finding out that the folder is an external storage mount and issue an extra request to ask some other property. But this is just shifting the job from the server to the client and complicating the whole protocol so maybe not the best idea.

Currently (2.1) the system is not consistent: it seems that quota-used-bytes is queried when user browses folders and oc:size is asked for in PROPFIND requests when discovering changes on the folders.

Please also, refer to this documentation: https://github.com/cernbox/smashbox/blob/master/protocol/protocol.md and this issue: #4455

@PVince81: your statement "The "quota-used-bytes" will retrieve the storage's used bytes, regardless which path you are querying." is currently true for our storage and the result looks like this:

So how comes it works for the owncloud server if your statement was true?

[icewind1991]

If there are mount points inside that folder (received shares, external storages), they are not counted in the size.

Incorrect, FileInfo->getSize takes submounts into account correctly

[icewind1991]

oc:size is used to discover the space occupied by the folder

That is currently the case

[PVince81]

@icewind1991 ok, thanks for the clarification. Was this always the case ?

[icewind1991]

Yes

[ogoffart]

Pushed in master for 2.2

[moscicki]

Thx!

I also updated the documentation accordingly (I hope this is compatible with current owncloud server implementation): cernbox/smashbox@ef70cbf

You may also want to think if you want to treat the external mount point in the owncloud server as a different quota node. I would say that you need to, because if a user had 1GB on the primary storage and 10G on the external mount, that 10GB free space is only available within that mountpoint (so on specific path). So if sync client wants to make a reasonable query about how much free space is left in the given directory before uploading to it, it should query that directory for available quota. Of course it can take shortcuts and understand that the quota information is valid for the entire directory tree associated with the quota node and not query all directories all the time. But for this you need to be able to discover mountpoints (i.e. different quota nodes).

This also brings to the next issue, which is that quota check should be done (and reported) on per sync-folder pair: imagine a user having two folder pairs: Documents (on primary storage) and Extended (mount on S3 storage). These are effectively two different quota nodes: there are two different quotas for these two different folders. It goes along similar lines as #4460

[ckamm]

I'm closing this after looking at the patch again and checking that size information on folders still works.