Persistent metadata + image cache (the pattern the other *arr apps use) #732
Paelsmoessan
started this conversation in
Ideas
Replies: 1 comment
-
|
Opened a working prototype as a draft PR: #733. It covers the image side end to end:
Running on a ~950-book library, that background job rebuilt the full set in about 7s with no browser involved. It's a draft for feedback, not a merge request. Happy to align the details with whatever you'd prefer here, and a metadata (non-image) cache would be a natural follow-up. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
The problem in one sentence
Right now, every time you open or refresh the library, Listenarr goes back out to the internet to
fetch cover art and metadata again, instead of keeping its own local copy.
That works with a handful of books, but it doesn't scale:
triggers rate limits or bans.
there's no memory of it, so the same work happens again on the next refresh. (We recently hit a
nasty version of this in Fix AudiobooksView freeze/leak and uncacheable cover 500s #731: a broken code path returned an error that browsers refuse to cache,
so the app re-requested it on every single screen redraw.)
How the other *arr apps solved this (checked against their actual source code)
They keep a local copy so they don't re-fetch from the internet each time. Two parts:
1. Cover images, the "MediaCover" approach (Sonarr / Radarr):
MediaCoverfolder (one subfolder per item).small image, not a giant one. They resize with the ImageSharp library.
There's no "remember that this one has no cover." That's a known annoyance (see Radarr issue #5711),
and it's something we could actually do better than they do.
2. Metadata, a shared "middleman" server:
that sits in the middle: Sonarr uses SkyHook (for TheTVDB), Radarr uses api.radarr.video (for TMDB),
Lidarr uses api.lidarr.audio (for MusicBrainz).
original source being slow, rate-limited, or changing.
metadata "became unusable." Books and audiobooks are the hardest kind of metadata to keep working,
so having a solid caching layer here isn't a nice-to-have. It's what keeps a project alive.
Where Listenarr is today
It fetches covers on demand and resizes them one at a time as they're requested; it looks up metadata
per request; and it doesn't remember misses. The good news: the app can already ask the server for a
small, grid-sized copy of a cover (built with ImageSharp, the same library the other apps use), so
we're partway there. The piece that's missing is keeping a durable local copy instead of redoing the
work every refresh.
Proposal
First (local, easy win): keep a proper on-disk cache of covers. Download and resize each once,
save it, serve it from disk, and remember when a cover doesn't exist so we stop retrying it. Warm
it up in the background so the grid is instant. This builds straight on the thumbnail work already in
place.
Later (bigger): a caching middleman for metadata, like the other apps use, so metadata isn't
re-fetched by every user on every refresh.
Questions
later?
Happy to prototype the local image and "no-cover" cache if there's appetite.
Beta Was this translation helpful? Give feedback.
All reactions