perf(labrinth/random_projects_get): speed up through spatial queries according to profiling results #3762
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…according to profiling results
Prospector
added a commit
that referenced
this pull request
Jun 14, 2025
* Show up to 15 projects in chart tooltips (#3739) * fix(frontend): remove fixed height from ManySelect (#2898) * fix(frontend): remove fixed height from ManySelect Frontend development is not my passion, there might be a better fix. I've tested my changes in all places that I found using the chganed components (ManySelect, ScrollablePanel): - Changelog filters - Version filters - Download dialog - Search filters Fixes #2334 * Revert incorrect merge * fix merge conflict * fix: reset reset-icon state value correctly in edit world modal (#3748) * Fix random_projects route not returning the requested number of projects (#3758) * Fix random_projects route not returning the requested number of projects * fix(labrinth): further improve random project route SQL query * chore: fix typo in comment * tweak(labrinth): more apparent and fast randomness for `random_projects_get` * tweak(labrinth): even better random projects query * chore: address formatting review --------- Co-authored-by: Alejandro González <me@alegon.dev> * Make get_user_from_headers and check_is_moderator_from_headers take in a bitflag of Scopes rather than a slice of Scopes (#3765) * Update a bunch of dependencies (#3766) * Add handling for new loaders, fix max height being applied when scrolling is disabled from #2898 (#3761) * perf(labrinth/random_projects_get): speed up through spatial queries according to profiling results (#3762) * fix: hydration issues caused by duplicate components on servers panel (#3753) * fix: server stats icons * fix: fix chart jumping * refactor: iconComponent -> icon * fix: panel hydration issues * fix: apply requested changes * fix: MOD-292 repair button showing during installation (#3734) * fix: MOD-292 repair button showing during installation * fix: lint * Update apps/app-frontend/src/pages/instance/Index.vue Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: IMB11 <hendersoncal117@gmail.com> * fix: lint issues --------- Signed-off-by: IMB11 <hendersoncal117@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * feat(labrinth): ignore email case differences in password recovery flow (#3771) * feat(labrinth): ignore email case differences in password recovery flow * chore(labrinth): run `sqlx prepare` * Add segmentation to reports list to fix it (#3772) * feat(frontend): Organisations are now sorted alphabetically in dashboard and on user pages (#3755) * feat: Organisations are now sorted alphabetically in dashboard and on user pages * Use computed ref --------- Co-authored-by: Prospector <prospectordev@gmail.com> Co-authored-by: Prospector <6166773+Prospector@users.noreply.github.com> * fix: undefined instance path by using emitted event instead when opening world folder (#3746) * fix: undefined instance path by using emitted event instead * fix: linting * refactor: Huge pyro servers composable cleanup (#3745) * refactor: start refactor of pyro servers module-based class * refactor: finish modules * refactor: start on type checking + matching api * refactor: finish pyro servers composable refactor * refactor: pyro -> modrinth * fix: import not refactored * fix: broken power action enums * fix: remove pyro mentions * fix: lint * refactor: fix option pages * fix: error renames * remove empty pyro-servers.ts file --------- Signed-off-by: IMB11 <hendersoncal117@gmail.com> Co-authored-by: Prospector <prospectordev@gmail.com> * frontend: Improve file too large error (#3774) * Improve file too large error Signed-off-by: IThundxr <me@ithundxr.dev> * MB -> MiB Signed-off-by: Prospector <6166773+Prospector@users.noreply.github.com> --------- Signed-off-by: IThundxr <me@ithundxr.dev> Signed-off-by: Prospector <6166773+Prospector@users.noreply.github.com> Co-authored-by: Prospector <6166773+Prospector@users.noreply.github.com> * enh(ci): optimize Turbo CI check workflow, track Rust and Node toolchain versions in well-known files (#3776) * enh(ci): optimize Turbo CI check workflow, track Rust and Node toolchain versions in well-known files * fix(ci): build `sqlx-cli` with `rustls` to fix Postgres TLS failures * fix(app): make instances with non-UTF8 text file encodings launcheable and importable (#3721) Previous to these changes, the app always assumed that Minecraft and other launchers always use UTF-8, which is not necessarily always true. * fix(app): make per-instance launch hooks clearable (#3757) * fix(app): make per-instance launch hooks clearable * chore(apps/app-frontend): fix Prettier lints * Update Rust and Turbo versions (#3781) * chore: bump Rust version from 1.86 to 1.87 * chore: update Turbo * chore(.cargo/config.toml): minor comment tweak * Small CI flakiness fix and performance tweak (#3780) * perf(ci): use Turbo to schedule both `lint` and `test` tasks at once * fix(ci): wait until service containers are initialized for tests This is achieved by adding a health check to the containers, and instructing the CI workflow to wait until the containers are healthy. Not doing this wait risks spurious CI failures due to DB migrations being applied before the DB even starts. * chore(turbo): use locally installed schema in new Turbo override file On the latest versions of Turbo, this ensures that the used schema is always in sync with what's available in the installed Turbo version, which is something that has already caused confusion to me before. * refactor: inherit Clippy lint config and Rust edition from workspace (#3782) * refactor: inherit Clippy lint config and Rust edition from workspace This also ensures developers running `clippy lint` locally get the same lints as during CI, especially when the Rust toolchain version is fixed through a `rust-toolchain.toml` file. * chore(clippy.toml): bump MSRV to 1.87 * chore(clippy): enable and fix many stricter lints (#3783) * chore(clippy): enable and fix many stricter lints These ensure that the codebase uses more idiomatic, performant, and concise language constructions. * chore: make non-Clippy compiler warnings also deny by default * Fix lint * Update changelog --------- Signed-off-by: IMB11 <hendersoncal117@gmail.com> Signed-off-by: IThundxr <me@ithundxr.dev> Signed-off-by: Prospector <6166773+Prospector@users.noreply.github.com> Co-authored-by: Erb3 <49862976+Erb3@users.noreply.github.com> Co-authored-by: Magnus Jensen <magnushjensen.mail@gmail.com> Co-authored-by: Emma Alexia <emma@modrinth.com> Co-authored-by: Alejandro González <me@alegon.dev> Co-authored-by: Josiah Glosson <soujournme@gmail.com> Co-authored-by: Alejandro González <7822554+AlexTMjugador@users.noreply.github.com> Co-authored-by: IMB11 <hendersoncal117@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: worldwidepixel <58098422+worldwidepixel@users.noreply.github.com> Co-authored-by: IThundxr <me@ithundxr.dev>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
From a functional point of view, #3758 is a fine improvement to the situation of the
random_projectsroute.However, today I decided to conduct additional performance testing using a synthetic table of 1 million projects with randomly assigned IDs. During this testing, I realized that the recently introduced
OFFSETclause in the related SQL query caused worse and unreliable performance, with execution times ranging from 50 ms to 500 ms, depending on the randomly generatedOFFSETvalue. Since the random projects route does not require authentication and the deployedTABLESAMPLEimplementation is very fast, this change will very likely be considered a regression in production, and place unnecessary additional load on our backend servers.This PR introduces the fourth (and what I hope will be the last) implementation of a query for fetching
nrandom projects. It significantly improves upon the previous one by eliminating the use ofOFFSETthrough another novel and elegant strategy:[min ID, max ID]range.nclosest projects to that point with a searchable status are retrieved. Each project is treated as a point on this 1D axis, where the coordinate is its ID. Since the IDs are assigned randomly, the nearest neighbors to a random point also constitute a random sample of projects.PostgreSQL's built-in first-class support for partial GiST indexes, combined with its spatial point distance operator, enable highly efficient query plans for this approach. In fact, this new query consistently executes in approximately 1 ms on the same dataset and environment, a >50x performance improvement. Overall, its total execution time for bigger tables is very close to that of the original
TABLESAMPLEimplementation, all while maintaining both correctness and code clarity (in hindsight, the algorithm implemented is quite straightforward to understand and prove to always yieldnprojects as long as there are at leastnvisible projects in the table).