Storage changes overview

[PVince81]

This ticket is here to gather and discuss the bigger picture or where we want to go with the storage/filesystem architecture. So far everyone is making small changes here and there so I think it's a good idea to have an overview to what we are working on now and also where we want to go in the future.

Here are some open ideas/tickets/PRs:

• deprecate file proxies in favor of storage wrappers (suggested by @icewind1991, needs ticket with plan/discussion)
• use OCP\Folder API in apps (and improve its performance): Internal apps should also use the file node-based public API instead of View #12471
• Jail_, Mask_, CacheWrapper: Add Jail and PermissionsMask storage wrappers #12426
• IChunkHandler idea: [WIP] introduction of interface IChunkHandler and a very first stupid implemen... #12160
• IVersionHandler, ITrashHandler for storage-specific version/trash handling: Move versioning and trashbin functionalities to actual interfaces on the storage level. #12274
• consolidate MappedLocal with Local (@nickvergessen)
• get rid of oc_filesmap: getting rid of oc_files_map #12461
• StorageFactory ?
• search/filter/sort on storage (ex: tags)
• storage specific options (encryption, version, scanner, etc): config option for external storage #12216
• custom cache impl (non-DB) ?
• storage ids / oc_storages redesign (+ legacy storage issues): [ext storage] Rely on numeric_id, store config in DB, improve config format #11261 Redesign/revamp oc_storages to fix various issues #11830
• ext storage config refactor: [ext storage] Rely on numeric_id, store config in DB, improve config format #11261
• provide ways to inject extra data in FileInfo /getDirectoryContents (CacheWrapper ?) for tags task
• redesign/rethink file scanner: Rework/improve the file scanner / file cache #11753
• reuse etags by default: Default to reusing etags in the scanner #12011
• consolidate file info attributes (from DB to JSON or WebDAV) to avoid several conversion steps ($data -> FileInfo -> json)
• move to exception-based errors instead of return false for FS functions: Storage API must work with exceptions, not booleans #13792
• high-level file locking: Implement high level file locking #11804
• refactor shared storage / cleanup: [WIP] Refactor shared storage #12086
• reuse known fileids and cache data in scanner: Reuse known fileids and cache data in the scanner #10993
• checksum support: Compute and store file checksums on the fly / file integrity #11811

Please add your own ideas about where you think the storage/filesystem part should go in the future to accommodate to new requirements (universal file access) and cleaner code/API.

[PVince81]

• make filesystem-setup non-static: Make Filesystem / setupFS non-static #12399
• do we want all files under "data/$user" stored in the oc_filecache ?
• metadata (thumbnails, versions, etc) storage for better quota management: Per app volatile metadata storage with quota #11963
• path handling tasks: Path handling tasks #8571
• move away from recursive path iteration: Review all recursive file operations - could cause issues on deep folder hierarchies beyond 100 #8476

[PVince81]

@DeepDiver1975 @schiesbn @nickvergessen @icewind1991 @MorrisJobke @craigpg @karlitschek

[DeepDiver1975]

• expose cache interface in public api for storage

[karlitschek]

Good list. Makes sense. A lot of work :-)

Some feedback from me:

• checksum is nice to have but could also wait for 8.1
• The storage options are really important. Specifically the different file-scanner behavior for different storage backends. (Cache refresh behavior)
• Maybe I missed it but we also need the option get direct download and upload links from the storage. (broker concept)

[PVince81]

• broker: Direct Download Link for Up- and Download - Was "Broker" #12286

@karlitschek the goal of this list is not "must implement everything" but rather "things to keep in mind when designing interfaces and refactoring classes".

[karlitschek]

@PVince81 Sure. But I think we can´t do everything at the same time. So it is good to agree an an order.
About the broker. I only wanted the be sure that it is not forgotten because it wasn´t on the list

[PVince81]

Feel free to edit the origin post and reorder/group them.

[DeepDiver1975]

• performance increase during webdav upload - Work directly on the storage when uploading over webdav #12006

[PVince81]

• Use "part" folder for huge folder operations: Use "part" folder for huge file operations #13756
• Use part files for web UI uploads: Use part files for web UI upload #13755 => achieved through PUT upload Use Webdav PUT for uploads #21237

[PVince81]

• move+update cache for every file one by one instead of bulk update: Move files + update cache one by one instead of bulk update #13775
• proper copy/move between storages: Proper copy/move between multiple local storages #13360

[PVince81]

• stat cache (storage wrapper?) for external storage to avoid too many requests: Improve ext storage performance with stat cache [$5] #7910

[PVince81]

We're making progress 😄 (ticked a few boxes)

[PVince81]

Ticked a few more boxes due to more progress ! @butonic @DeepDiver1975 add yours...

[butonic]

I am trying to store metadata via extended attributes, which makes me wish the primary key of the table was (storage, fileid) and we were referencing the storage id along with the file id throughout the code. They map to device id and inode pretty well. If we touch the filecache table we should collect other requirements. Currently it is impossible to resolve the owner to a fileid by using sql. The oc storages table might contain a hashed id which causes us to iterate over all user for certain file actions like trash and versions expiry (instead of looking at the files directly).
Things that come to mind:

• use bigint for fileid
• rename oc_filecache to oc_fileindex because it is NOT a cache: the content cannot be rebuilt without losing the fileids & etags.
• rename storage to storageid
• make (storageid, fileid) primary key ... and use this combo throughout the api
• remove unused columns (encryption no longer needs all of them AFAICT)
• Instead of doing the migration during upgrade we can create the new table and do a user by user migration?
• store files as part of the files app: data/files/$user?

[PVince81]

Currently it is impossible to resolve the owner to a fileid by using sql.

You can now use the oc_mounts table to reach it, see #1344 (comment), but still not very clean...

[PVince81]

• cleanup storage PHP interfaces/classes: Cleanup storage interface #28996

cc @jvillafanez

[PVince81]

• decouple storages from users: Decouple home storage entry from user #29502

[PVince81]

ticked a few more boxes, baby steps 😄