-
Notifications
You must be signed in to change notification settings - Fork 168
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Extend caching plugin to handle markdown and iptc metadata #443
Conversation
Codecov Report
@@ Coverage Diff @@
## main #443 +/- ##
==========================================
+ Coverage 87.33% 87.93% +0.59%
==========================================
Files 23 23
Lines 1943 2014 +71
==========================================
+ Hits 1697 1771 +74
+ Misses 246 243 -3
Continue to review full report at Codecov.
|
I have an album with 150K pictures and 4K videos and use the extended_caching plugin. The 2.2 release takes approximately 18 minutes to rebuild the gallery with the plugin enabled (assuming no thumbnails to generate) and the cache file is approximately 23MB. Using the code in this PR it takes less than 5 minutes to rebuild the gallery with a cache size of 1.03GB (43x the previous size!). The pickle.load call in my setup alone takes 60 seconds of CPU time! While that is a little crazy the net effect is a 60% reduction in wall time to complete a full rebuild even with the previous extended_caching in 2.2. It is faster to parse the single file than to read 154K files for metadata across the filesystem as currently done in gallery.py -> class Media -> init -> self._get_metadata() |
Sorry it seems I forgot to review this PR. I just had a look and it looks very good. |
…s updating when a file timestamp changes
d08cf71
to
f2451f2
Compare
@saimn should be good now |
Excellent, thanks for the quick update ! |
Turn markdown metadata in
gallery.py
into a property so it can be overridden.Extend the caching plugin to handle markdown and iptc metadata (previously only handled exif).
Invalidate cache when a file timestamp changes, so we always get the latest data.
Cache file modification time lookups within a build to reduce load on filesystem.
Supersedes #440.