Trashbin on external storage

[oktawave-code]

Hi,
we use external storage (via swift) to keep all files. The problem is that the trashbin works independently, and stores files locally. This cause several issues:

• data must travel form external storage to local storage and vice versa when deleted/undeleted
• our local storage is much smaller than dedicated external storage, so the local storage is insufficient to keep all trashed files
• we need to have at least 2 webservers that will serve owncloud web interface, the http traffic will be balanced between them. So we want to avoid keeping any local files on webservers, all data (users files) must be stored externally

My question is: will the trashbin support external storage just like other parts of owncloud in the nearest future? Is there a simple way to add a support for external storage in current oc version?

Best Regards,
Tom

[DeepDiver1975]

@schiesbn

[RobinMcCorkell]

Bumping - I will have a go at implementing this at some point.

[PVince81]

@Xenopathic can you describe your ideas here as discussed today, you referred to the way how .gvfs was doing it 😄

[RobinMcCorkell]

@PVince81 you're having fun mentioning me in every issue, aren't you 😆

The way GVFS (Gnome Virtual FileSystem) works is, in my opinion, very sane. It provides a trashbin, similar to oC's, but since it is running on a real filesystem it puts 'deleted' files into .local/share/Trash along with the metadata for those files. On a mounted filesystem however, like SMB/CIFS, it doesn't try to move files across the filesystem boundary, instead creating a .Trash-<uid> directory in the root of the remote filesystem that is analogous to .local/share/Trash for the local filesystem. <uid> is of course replaced with the UID of the user to prevent trash conflicts on shared storages.

A similar thing could be done with files_trashbin. I suggest having a directory in the root of the external storage, .Trash-oc-<username> for example, which will contain the 'deleted' files. To maintain database and filesystem consistency however, the metadata must also be stored in this directory, not in the database as it is at the moment.

The only issue that needs resolving is encryption. As far as I know, when a file is put in the trash bin its encryption keys are also put there. Again, to maintain consistency this means those keys will end up on the remote storage - where they could be accessed directly by users. As long as all encryption keys are encrypted with a key only available on the ownCloud server, this point is no longer valid, but it needs to be checked for security.

[PVince81]

We could also have a single shared trashbin on the external storage, this way multiple users have the possibility to restore old files and see the same deleted files (suggested somehow here #7676)

[tibmanus]

I think this would be the right issue to express my confusion about the space management.
The documentation tells me the following: "Deleted files that are still in the trash bin do not count against quotas. The trash bin is set at 50% of quota." But what is the quota, what if I don't use a quota for my users?

Here is an example about my confusion:
I have the server application installed on a disk with limited capacity, so I mount up locally a very large drive as an external storage and I only keep my files here. When I delete files, all these files are copied back to the trans on the local storage and I see that my total available space has reduced. I can keep doing this until I reach 0byte availability. What happens then? Why the versioning/trash didn't get automatically emptied? How should I read the "50% of quota" in this case?

[PVince81]

You're right, it's not very clear. The quota value is the one of the home storage and doesn't account for any external storages. Making the quota count external storages is tricky because it would be different depending on the folders.
Also the trashbin + versions information (the deleted files and version files) are stored in the "data" folder, which isn't currently on external storage.

This is why it was suggested to have trashbin and version stored on the respective external storages. The expiration logic could then be run for every storage separately based on the space available on each storage.

Hope that makes sense.

[PVince81]

So in your case if your "data" folder doesn't have a lot of space, then versions and trash might expire very often.

[tibmanus]

So reaching 0bytes will still give me a working cloud, but without version/trash?

[PVince81]

Yes, without versions and trash.

[SergioBertolinSG]

I find this very interesting to have since currently in 8.2beta1 it isn't possible to delete a group of files in external storage => annoying.

[PVince81]

currently in 8.2beta1 it isn't possible to delete a group of files in external storage => annoying.

@SergioBertolinSG can you elaborate ? Maybe you mean #16533 ?

[SergioBertolinSG]

Yes, exactly that thanks.

[PVince81]

If you want to test deletion, you can create a subdir. That problem only exists for files directly under the mount point.

[rr-paras-patel]

I use AWS S3 as primary storage and implemented .files_trashbin and .files_versions folder inside root space of bucket similar to .Trash-oc- you discussed previously but problem is that in case of AWS-S3 they don't have object rename method or API officially because its object storage or business policy of AWS so in order to rename or Delete object(move to trashbin) i need to pass following steps

1. Copy object from orignal into (.files_trashbin/filename.dtimestamp)
2. Delete orignal file

Step 1 in case of big size like 1 GB++ and if files have some versions also then move everything to .files_trashbin and .files_trashbin/version folder it takes long time to copy. and i can't imagine scenario incase of large object 8 GB ++. better to provide software level object rename or maping (DB:files_cache) table need to include object mapping column. so we don't need to move or rename object on storage just update object name in SQL DB. I know this steps can increase complexity in user's quota space count method.

Reference of rename method in AWS S3 : https://github.com/owncloud/core/blob/master/apps/files_external/lib/amazons3.php#L514

[nicoolas]

Any news on this topic ?
Using version 9.1, I just deleted a large amount of data from an external storage, that ended up downloaded on my server, and filling up disk space.

Otherwise, anyone knows if trash can be disabled altogether for external storage ?

Thanks

[plastikschnitzer]

If you delete files directly on the external storage aside ownCloud, it won't fill up your ownCloud trashbin.

For some users this might be desired behavior, for others not …

[nicoolas]

It sure could be a desired behavior;
However, I think that an option to disable the trash bin for external storage would be appreciated - simply because external storages are quite often setup to limit server disk usage.

[RedKage]

Hello. Any news on this? I stumbled on this issue because of an error when I was trying to delete files from my G Drive. Possibly, this issue here is related.
/subscribed