Lazy-materialize stats-page filter combos into stats_summary#385
Merged
Conversation
The stats page lets users pick character + win + ascension + players
filters, but HOT_FILTER_COMBOS only materializes {} + per-character.
Any other combo falls through to the live aggregation, which runs
several pipelines (totals / ascension / deaths / pick_rates with
$unwind / etc.) and takes 5-10s.
The process-local TTL fallback cache is per worker, so with 4 workers
the cache hit rate on a clicked-through filter sequence is poor and
users see a 5-10s spinner on practically every filter change. That's
what was being reported as "stats page settings breaking."
Add a write-through step: after a live aggregation runs in the route,
write the result back into stats_summary keyed by the same filter
tuple. Subsequent requests for the same combo -- on any worker, ever --
read from stats_summary in a single find_one (~ms).
- First request for any filter combo: still ~5-10s (one user pays).
- Every subsequent request for that combo, cluster-wide: sub-ms.
- HOT combos keep being refreshed every 60s by the existing refresher
loop and overwrite any lazy entries with fresh aggregates.
- Non-HOT combos persist in stats_summary; they'll get overwritten the
next time someone queries them with new data. If unbounded growth
becomes a concern later, add a TTL index on updated_at.
ptrlrd
force-pushed
the
perf/stats-lazy-materialize
branch
from
84b2122 to
384a7be
Compare
This was referenced Jun 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Symptom
Users picking filters on
/leaderboards/statssee a 5-10s spinner on practically every click. The same combo on the next click might be instant or might be slow again, seemingly at random.Cause
/api/runs/statshas a 3-tier read path:stats_summary(materialized, sub-ms) -- only coversHOT_FILTER_COMBOS = [{}, per-character].--workers 4, so the hit rate on any clicked-through combo is at best 1-in-4.get_stats()-- multiple pipelines including a$unwindovercard_choices. 5-10s per call on the currentrunssize.So any combo with
win,ascension,players, orusernamefalls past step 1, and step 2 is poor cluster-wide because the cache lives in process memory. Result: clicking filters keeps paying the 5-10s tax.Fix
After the live aggregation runs in
get_community_stats, write the result back intostats_summarywith the same(_filter_key)shape the refresher uses. Now every worker (and every future request) reads the same materialized doc.find_one, ~ms.stats_summaryand get overwritten the next time the same combo is queried. If the collection grows uncomfortably (it's small per doc), follow-up: add a TTL index onupdated_at.Files
backend/app/services/runs_db_mongo.py-- new publicwrite_stats_summary()helper.backend/app/routers/runs.py--get_community_statscalls it after the live path.No data migration; no schema change; falls through to existing behavior on any failure.