Restrict occ index command to specific folder

[terencedesu]

Love the Memories app. Set it up, indexed my instance and set my directory path, but for some reason, if I add new pictures to my photo collection (which is on an external drive) they don't show up in the Memories app. I tried running the index again, but I have since added several more drives and the index command wants to scan everything. I haven't been able to successfully scan the instance again, but also don't want to.
Do I have to index each time, and if so, is there a way to restrict it to just my photo collection?

[pulsejet]

Ideally you should be adding .nomedia to the folders whose trees you don't want to index.

[terencedesu]

Thanks for the quick response. I thought I had done that for the drives I didn't want to index but maybe I missed it. I'll give it a shot.

[terencedesu]

Turned out I needed to add the .nomedia to a few more folders. Thanks. Followup question. Do I always have to do the scan command or is there a way to add photos to the nextcloud instance and not have to run the command to have them show up in Memories.

[pulsejet]

In general, you should never need to re run the command except:
1/ When you upgrade (and the changelog indicates you need to)
2/ When you change nomedia files and you care about the folders shown (for now, at least)

[Glandos]

Ideally you should be adding .nomedia to the folders whose trees you don't want to index.

That's one mechanism. But on my Nextcloud storage I have:

• A lot of documents directory
• One particular photo/video directory on an external storage, in a subdirectory

I need to put a .nomedia everywhere else? This would cripple my own computer's directories synchronized with Nextcloud 😞

[pulsejet]

Is there anything you're trying to achieve other than reduce initial indexing time? Indexing everything shouldn't have any impact on performance. As long as you set your timeline path to the directory with your photos, it should be fine.

[Glandos]

You're right, it's not that important.

Other interesting point: database can be a bit lighter when not containing uninteresting files. On my setup, I have 68k records, and an quick estimate of 10% (6k) are irrelevant.

[pulsejet]

Other interesting point: database can be a bit lighter when not containing uninteresting files. On my setup, I have 68k records, and an quick estimate of 10% (6k) are irrelevant.

I can't imagine that being more than a few megabytes of storage extra. As I mentioned, this shouldn't have an impact on performance.

[vipseixas]

The "index all, show some" behavior (#22 (comment)) has a major problem.

My situation: I have about 400k files and 90% of them are in S3 Glacier Instant Retrieval, which means that they are meant to be sleeping there and it's expensive to access them. The normal pattern is that old files are not accessed often (less than 0.05% each day).

I could spread .nomedia files across my directories, but I can't control other users directories.

[pulsejet]

@vipseixas what behavior do you suggest then? I'm not sure what makes sense if you don't have control over how users store files / photos.

[vipseixas]

I think the index command should only act on the directories that are configured as memories root for each user or, as @terencedesu suggested, have a parameter to configure where the index process should start.

Another idea is a parameter to limit the amount of files to index for a single run, so I can control how many files will be indexed per day and plan the indexing costs through time.

Later I'll browse through the code to see how the index is done. Is there a technical limitation to a partial index?

[pulsejet]

I think the index command should only act on the directories that are configured as memories root for each user or, as @terencedesu suggested, have a parameter to configure where the index process should start.

We can't scan the timeline folder selectively because in typical usage the indexing is done before the timeline folder is selected, plus the timeline folder may change. But having a parameter for the directory path to scan is a fair idea.

I want to note that this still doesn't work if you have absolutely no control over how users store data. At the minimum, you still do need to tell users to place their photos in a certain directory.

Another idea is a parameter to limit the amount of files to index for a single run, so I can control how many files will be indexed per day and plan the indexing costs through time.

I'd be wary of this on a production system; users would see half their library of photos and think something went wrong.

Later I'll browse through the code to see how the index is done. Is there a technical limitation to a partial index?

No limitation other than what's already mentioned. I'm open to a PR implementing indexing only a fixed path, provided as a system-wide config option (we need this to restrict scanning for future uploads in PostWriteListener).

[vipseixas]

At the minimum, you still do need to tell users to place their photos in a certain directory.

That's the idea, I'll announce to the users (basically family and some friends) which path they must use to have their files indexed.

Another idea is a parameter to limit the amount of files to index for a single run, so I can control how many files will be indexed per day and plan the indexing costs through time.

I'd be wary of this on a production system; users would see half their library of photos and think something went wrong.

This could lead to undesirable side effects, for sure...

I'm open to a PR implementing indexing only a fixed path, provided as a system-wide config option (we need this to restrict scanning for future uploads in PostWriteListener).

Great, I'll test this as soon it's done. Thanks!

[wonx]

I'd also be interested in this. I only have like 4 folders with pictures, but occ memories:index is indexing all the files in the filesystem and it's quite slow. At the moment the process has been running for 11 days, and it's still at the 19.4%, so I can expect the whole indexing process to take around two months.

I'd rather not fill my directories with .nomedia files. So, the same way occ files:scan has the --path parameter, it would be great if memories would have something similar, so I could just schedule a scan for these 4 dirs with photos.