Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

jobs endpoint - high cpu / memory query #442

Open
ITJamie opened this issue Jun 21, 2022 · 0 comments
Open

jobs endpoint - high cpu / memory query #442

ITJamie opened this issue Jun 21, 2022 · 0 comments
Labels
bug Something isn't working

Comments

@ITJamie
Copy link
Contributor

ITJamie commented Jun 21, 2022

Describe the bug
the jobs/filter endpoint gets loaded when viewing the jobs page. the user_list query is unoptimised
When you have a jid's table with a few 100k rows it can bring your machine to a crawl.
the current code has to load every single row into memory. json parse it in python then find all unique "users" across every dict.
while this is fine in a dev environment, it can consume gigs of memory on a large db.

To Reproduce
create or import a very large jid's table.
load the jobs page in alcali a few times. watch memory usage on postgres + python alcali gunicorn

Expected behavior
not consuming multiple gb of memory

Additional context
I would recommend removing the user_list filter on this endpoint OR at least giving a config option to disable it.
The jid's table does not have a username column to filter on by default, any searching of it is going to cpu and memory intensive on larger dbs

@ITJamie ITJamie added the bug Something isn't working label Jun 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant