How to deal with empty parent folders?

[oparoz]

When building the gallery view, we need to provide navigation so that sub-folders containing media files can be shown as albums.
We also have to make sure that looking for media files doesn't "freeze" the app.

So how do we address the case where pictures are only found 10 levels deeper in a large repository with many folders per level?

Some thoughts

• Drilling down each folder to see if there is something to show could lead to the issues we're currently seeing when trying to load the whole hierarchy in memory when first loading the app.
• Stopping at an arbitrary number of levels could lead to empty galleries, even though there are media files to show.
• Defining only one folder as the "Gallery folder" doesn't work for most people as it requires organising all media files, many of different types in a single folder.

Any suggestions @icewind1991 @libasys @ftree @bwoester @jancborchardt @dbeckmann @TheDD?

[bwoester]

Searching the filesystem when retrieving data is generally a bad idea. It's just too slow, even in the case of a local filesystem. Now, think about what would happen when using a remote file system.

Maybe it's worth considering how difficult it would be to approach the problem from the other side. What would need to be done to know where media files are stored instead of having to search for them?

I'd probably go with some sort of a prepared data provider/cache. This could be a db table or a document store collection, providing the information necessary for the view. File ID, location, mime type, file size, image resolution, video codec, ... you name it.

I say "data provider", because this component would be the single source of truth for the gallery view. No more searching for files in the filesystem. I say "cache", because the data provider can easily be cleared and re-filled. The data is available in the filesystem, this component only provides fast access to said data.

Now, the main problem becomes: how to keep the data provider in sync with what data is available?

One easy way is to just clean it, search the filesystem and re-fill it. This could be done as a whole or on a per folder basis, depending on the data provider's interface. Sort of a refresh action. The important thing to note is, that this operation can be decoupled from the actual query. It can be done in advance, so when the view starts its query, the data is available. The refresh could also make use of some optimizations, using files mtime to skip processing files that didn't change. Maybe there is even a way to detect if a folder's content changed, so it can be skipped altogether?

A way more effective approach however - although I don't know if owncloud's core provides such a feature - would be notifications. Imagine the component would be notified whenever a file is added, changed or deleted. It could simply remove the entry for that file from its persistent data store or extract the necessary data and add/update an entry for that file.

[oparoz]

@bwoester - I agree, but I thought all those core methods were already using the filecache stored in the database. Someone familiar with the filesystem layer should pitch in.
Regarding notifications, it's possible to listen to such events as addind/removing files, but that hasn't been implemented in the app yet.
And on oC7, it was impossible to use the Gallery app with external storage as it had too many problems and was way too slow. I haven't tried on oC8 yet.

[oparoz]

The current solution seems to be working, so I'm closing this.