Repository navigation
Home-timeline fan-out on a multi-threaded Redis-compatible server: up to 2x throughput, half the timeline read latency (with Mastodon's gems tested) #41702
murtazayusuf
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
On a busy instance, Redis's ceiling is home-timeline fan-out: every new status becomes a handful of commands per follower (
add_to_feed+triminFeedManager), all queued on Redis's single command thread while the web tier reads timelines from the same server. Once that core is saturated, adding Sidekiq threads doesn't deliver posts any faster; it just makes timeline reads wait longer.Disclosure: we build KiviDB, a Redis-compatible server (AGPLv3) that runs commands on all cores. We wanted to know whether it actually helps Mastodon, and whether it's safe to run under it, given #35863 (Stoplight's Lua returning a string on DragonflyDB, bolshakov/stoplight#414 — cc @ClearlyClaire, who tracked that one down). Sharing the results for admins weighing the same question.
Home-feed fan-out
20,000 home feeds at
MAX_ITEMS(800 entries each), driven with the exact per-follower command mix fromFeedManager#add_to_feed/#trim(ZSCOREon the reblogs set,ZADD,ZREMRANGEBYRANK 0 -801,ZREVRANGE 40 40,ZRANGEBYSCOREon reblogs,EXISTS subscribed:timeline:*) plus the home-timeline read (ZREVRANGEBYSCORE … LIMIT 0 20). memtier_benchmark, 4 threads, 30 s per step,redis:8(8.10.3) vskividb/kividb:v1.2.2on the same 8-vCPU host, each started fresh, two runs each:Redis is flat from 48 connections up: that's its one core. KiviDB keeps scaling with load, so the same server delivers up to twice the feed inserts while home timelines load in under half the time.
Compatibility, at Mastodon's
Gemfile.lockversions-luabuild)sidekiq.ymlqueues, 25–1,000 threadsThe sidekiq-unique-jobs differences: the Web "delete all locks" action sends
ZSCAN … COUNT 0on an empty set, which Redis answers and KiviDB rejects, plus a test-only fixture using Lua 5.1's implicitarg. The lock/unlock/reaper scripts all pass. Filed as kividbio/kividb#820.Trade-off to know: KiviDB currently stores these feed sorted sets in about 50% more memory than Redis (1.86 vs 1.22 GiB for the 16M entries above), so size the box for that.
Happy to share the scripts, or to run something closer to a real instance's traffic if anyone has a shape in mind.
All reactions