Jellyfin
Jellyfin Database SQLite vs PostgreSQL & Valkey #17211
Replies: 5 comments 19 replies
|
Couple of basic overview questions : 1- With 12.0 Release, does jellyfin now support PostgreSQL as a replacement for sqlite and thus offer improvements in performance with large libraries and large cocurrent users. 2- Has there been any shift in the sentiment to support postgreSQL now as a replacement for sqlite? My understanding is that sqlite is a major bottleneck in this scenario and a full DB like PostgreSQL will eliminate the bottlenecks caused by sqlite |
|
Hosting large libraries 100k+ libraries with 100+ cocurrent users is a pretty in scope thing for jellyfin. Another thing which idk if its missing or not, im not much familiar with jellyfin, is AI support in JF server and client. Like when searching, it should intelligently get the movie name, writing half words etc. AI recommendation and search is a major feature that id like to have in JF since it makes it makes the experience just so much better |
|
I couldn't agree more. Personally I only have about 5 users in my family but I've had serious issues with concurrency causing serious faults in the database. I couldn't agree more about the other AI features but from what I see in the release notes, none of this is on the horizon. Version 12 Release Notes: |
Jellyfin is a private home media server, not a commercial product. Supporting this kind of load is NOT our focus at all. |
|
Hi, what are your thoughts on the body and information of the post? |
|
From the release notes in 10.11: https://jellyfin.org/posts/jellyfin-release-10.11.0#plugins
Unofficial test plugin written by me using this new interface for postgres support: As ghost said, jellyfin is a home media server and we have no desire to cater to commercial usage by spending our freetime to allow somebody else to profit off it. So our target will always be just supporting a handfull of users and parallel streams. |
|
This sounds overly complex for the vast majority of Jellyfin deployments (has jf ever stated a goal of regularly supporting 100+ concurrent users!?). In the reality where basic features like LiveTV are breaking in new ways every release, prioritizing complexity for the sake of edge case scaling use cases seems misguided. An "ORM and choose" approach would prob get everyone close to what they want but from previous discussions that sounded like a significant effort as well. |
|
Some of what you're attributing to me are things I never said.I have exactly five users excluding myself & the admin account. My immediate family. Just last week I had to go in and manually patch the SQLite database because of the well-known concurrency issues. SQLite doesn't work well with this workload. The data I've showed above is to show that this is a majority case. If you hdave ever re-scanned a music or photo library in Jellyfin? It basically says "see you next year." This isn't about some massive, exotic install. Once you have any real number of items, Jellyfin becomes tremendously slow, especially on ZFS, and everyone is on ZFS. There's already a project that does exactly this and asking for some of that code to be merged is part and parcel of how Github works, so it's not as big an ask as some of the more out there requests. Let people dump their data into Postgres and Valkey instead of SQLite. And if you've never set up a Postgres database. it's the easiest thing you'll ever do. Easier than setting up Jellyfin, by a mile. It's a hard thing to recommend Jellyfin when you've got a little niggle at the back of your mind saying 'yeah but it doesn't really work does it'. The worst part is that it's an issue that you don't realize until you're fully invested, after you add your books and comics etc. |
|
I think there is a sensible middle ground here. On one hand, I completely agree with the collaborator (@Shadowghost) that designing Jellyfin to support 5k to 10k concurrent users is out of scope. Jellyfin is first and foremost a private, home media server. Bundling a heavy database like PostgreSQL by default, or rewriting core structures just to support commercial-scale traffic, doesn't align with the project's focus of being easy and lightweight to deploy. SQLite is the perfect default for 95% of users. On the other hand, the filesystem conflicts (like SQLite WAL on ZFS/Btrfs) and database locks can be a real friction point, even for small family deployments. Providing the optional capability to hook up an external database seems like the ideal compromise. It gives power-users the freedom to use PostgreSQL or Valkey if their hardware or library demands it, without cluttering the simple out-of-the-box setup for everyone else. |
|
I can agree with that. Having the option for an external database feels important at this point in technological development. I'm not with UsmanTariq2 in the literal sense but in the figurative sense of pushing software to be better & more scalable I am. I've seen it become standard even in less-supported Docker containers for applications of this type. Immich, for example, expects an external PostgreSQL instance with Valkey but will fall back onto its own SQLite DB. Same goes for Mealie, Linkwarden, UptimeKuma. At this point having the option is almost an expectation rather than a nice-to-have. And that's before you factor in the underlying problem of trying to grow an SQLite database on top of copy-on-write. The workloads of those mentioned other than Immich are vastly lighter than Jellyfin. |
|
10k figure was always aspirational. Lets reduce that down to 500 cocurrent users theoretically with 500k + piece of content. I think thats a decent target architecturally. |
|
A sensible upper bound is like 20 concurrent users at the high end. It's a private home media server, not a FOSS Netflix. |
|
To be honest, I feel like I’m doing something wrong when using Jellyfin. |
Immich is indeed quite powerful. I tried migrating those messy media files (not 60k,just 10k) to it a few months ago, and so far, the results are very good. |
|
I tend to agree with your point. I have a library that mixes images and videos, containing about 60,000 media files. |
|
for 500 concurrent users, not even a gigabit fiber connection is enough with only now becomes slowly the standard in most countries far less is still standard and how you fit 500 people into a single building is beyond me. Our target is a single home with the occasional remote stream while on the go. What kind of GPU could support 500 realtime transcodes of 1080p media? |
|
Once again, yall are missing the point. it could be 100 cocurrent users theoretically. Im arguing from a design perspective. And that number is aspirational. But , ur missing the thread of argument, ur coming at it from a weird angle tbh. The number is just aspirational, i have switched between a big range here. It doesnt matter what number it is, but a big enough number to never cause any issues, both for cocurrent architecture , but also library size, library management, user management etc. People invest alot into their media libraries. Libraries get big. One would assume jellyfin caters to the upper end of library collectors instead of arbitrariliy limiting itself to the lower bound of users. That goes against design principles of developing for upper bound of performance and users |
|
then please make the effort to explain it to us. We told you, everything above a single household worth of users plus some is simply out of the scope for JF and external DBs are already supported with 10.11 as i explained in another comment here https://github.com/orgs/jellyfin/discussions/17211#discussioncomment-17751223. So yea i do not understand your point. |
|
Perhaps instead of investing so much time and effort into building your library you could invest in watching it. |
|
You guys are being a certain way right now. It's clear that he means to build it well. And know he's not speak about commercialization. The fact that he's using the number 500 is just that it's an abstract big number, and you guys are all smart enough to know this. 😐 I'm certain that my instance of Immich can handle 500 people scrolling through the main timeline, sharing images, watching videos at the same time 🙂. It's not commercial software, either. Choosing to be obtuse pushes conversations into a death spiral. I think we all have the responsibility of not doing that and taking people in good faith. Please 🥺, less arguing more team work. |
|
This might be a nice middle ground when it hits a stable version https://github.com/tursodatabase/turso. I agree though supporting postgres would be nice. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Rethinking Jellyfin's Database: SQLite, Copy-on-Write Filesystems, & Performance at Scale
Summary
After several years of testing Jellyfin, I've found a class of problems that I believe are causing downstream pain for users and server administrators without getting much attention. The most significant one, in my view, is the database choice - specifically the way SQLite's write-ahead log interacts with copy-on-write filesystems. This post lays out what I've observed, the testing I've done, and a proposal for how Jellyfin might address it.
The Core Issue: SQLite on Copy-on-Write Filesystems
To be clear up front: I'm a big fan of SQLite. I think it's a smart and effective database for Jellyfin, especially given the project's ethos of allowing quick and easy installation. The problem isn't SQLite itself - it's an interaction that has largely gone unnoticed.
From what I can see, the majority of people installing Jellyfin are doing so on Linux-based operating systems, and a significant portion of them are using copy-on-write (CoW) filesystems. SQLite's write-ahead log (WAL) conflicts with the copy-on-write nature of these filesystems. For smaller installations, this overhead is tolerable. For larger libraries, it changes drastically.
Many of you have probably seen the complaints from people with libraries holding over 2,000 unique items, reporting that search query latencies have climbed above five seconds. Part of this is due to under-the-hood changes after version 10.7, but the filesystem interaction is a major and under-discussed contributor.
Testing and Results
I've seen significant performance gains simply by moving my config directory from a ZFS-backed NVMe SSD to an ext4 NVMe SSD.
To isolate the database specifically, I took the exact path containing the Jellyfin database and moved it onto a small 16 GB Intel Optane drive. On that drive, I tested three filesystems: Btrfs, ZFS (with suggested changes for SQLite), and ext4.
The results were stark:
The pattern is consistent: the copy-on-write filesystems impose a large latency penalty on database operations, while ext4 does not.
Proposal
Based on all of this, I think now may be a good time to reconsider how Jellyfin handles its database. A couple of options:
-1. Putting the current Jellyfin database in its own subfolder so in order to put it on faster storage the entirety of plug-ins need not be remapped.
Either approach would let administrators with larger libraries - or those running on copy-on-write filesystems - avoid the WAL/CoW conflict without abandoning the simplicity that makes Jellyfin easy to install in the first place.
All reactions