Import performance issues

[vkalinsky]

Hi everyone,

Is it possible to "gradually" import pictures/videos?

Currently PP performs all steps at once, like getting metadata, making a thumbnail, converting, making optimized/scaled image, labeling images with TF, etc etc.

So indexing the whole library of ~40K pictures takes forever, which would be totally fine if performed progressively. This way items would become available to view right away, then come the thumbnails, then TF labels etc etc.

I thought I was doing something wrong or misconfigured something, because I read somewhere in the issues that the dev team routinely indexes 150K pictures everyday for testing purposes. But the code is clear:

		if !Config().DisableTensorFlow() {
			// Image classification via TensorFlow.
			labels = ind.classifyImage(m)

			if !photoExists && Config().Settings().Features.Private && Config().DetectNSFW() {
				photo.PhotoPrivate = ind.NSFW(m)
			}
		}

Any suggestions?

The only idea that I could come up to is to disable TF, index the library, then re-index it with TF enabled. That way indexing will take 2-3 days on my NUC instead of 2-3 weeks :-)

[lastzero]

Any reason you import 40k pictures instead of indexing the directory they are in currently? Import will copy them to the originals folder and then delete the file in the import folder, so it's more I/O demanding than just indexing. Note that you can also import / index specific folders by selecting them in the dropdown, or providing the folder name as CLI argument. This way, you can do it in batches. If CPU load is too high for you in general, you can reduce the number of workers via config option (see docs).

[vkalinsky]

Thanks for the reply! I don’t actually import them, I ro mount the “originals” folder into PP docker container and let it index the folder.

[lastzero]

OK, so you can select and index sub folders step by step instead of indexing all files in one go.

[vkalinsky]

I'm not (yet) familiar with the project, and I don't quite understand how manual indexing solves the problem. Like I can add items quickly without full parsing/converting/labeling, then select and index folders step by step?

My suggestion was to put all those tasks into a priority queue, so items would become ready to view instantly, right after file metadata is added to DB. Then, after all 100 or 40000 files added, run labeling/nsfw/etc. At first glance it's not that easy to implement because it would require to spread this tiny result = ind.MediaFile(f, opt, "") over multiple tasks in a queue, deal with event publishing etc.

[lastzero]

Would probably take much longer as the same data has to be loaded multiple times. Also there may be issues we with stacking and duplicates while files are in an incomplete state.