Performance degradation based on library size

[DrazorV]

Hello! I will try to document my real world experience to better help understanding the performance impact especially as libraries grow. Please fill free to correct me if something is not accurate or should be mentioned in a different issue.

Setup

• Synology Nas 723+ (AMD R1600 and 18GB ram)
• Manyfold on a docker-compose stack with redis7 and pg-16

Library

mostly stls and 3mf but has images and lys files too

Loading Times

• Home Page: 500ms
• Settings: 115ms
• Creators: 6.39s
• Collections: 6.50s
• Library Page: 31.71s

CPU/Ram Usage
Everything below is documented while I perform a scan (not the initial scan)

Manyfold Conainer

Postgres Container

Redis Container

Additional Info

• Manyfold container has 12gb ram limit (never reached)
• Redis container had 256mb limit that I had to remove because it kept shutting down
• Noticed some ram usage increase when an additional user logged in (from 2.1GB to 3.5GB)
• Pagination (number of elements per page) does not affect loading time for the Library Page tried with 2, 12 and 50 elements.
• "Create tags from model directory name" setting is enabled
• Activity page doesn't load at all (it did work for couple of minutes but stopped loading after the scan progressed)
  

Since pagination doesn't affect loading times, it safe to assume that the performance hit doesn't rely on the number of models per page nor the type or size of models and their previews.
After some discussions we came to the conclusion that it could be affected from the number of tags created which is close to 16000 for me

and/or the number of database queries.

If there is anything else I can provide please ask!

[Floppy]

This improved about 33% with the N+1 fixes in #2085 , which is a start. After more investigation there's definitely some slowdown in the building of the tag list in ModelFilters, which ends up doing a lot of queries, and in this case with 16000 tags, I mean a LOT of queries.