Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve api/activity/recent_items performance #43092

Open
1 task
Tracked by #35564
escherize opened this issue May 23, 2024 · 3 comments
Open
1 task
Tracked by #35564

Improve api/activity/recent_items performance #43092

escherize opened this issue May 23, 2024 · 3 comments
Assignees
Labels
.Backend .Performance .Task Not a part of any Epic, used by the Task Issue Template .Team/AdminWebapp Admin and Webapp team

Comments

@escherize
Copy link
Contributor

On stats, it is ~500ms but hopefully should be ~200ms.

e.g.: /api/activity/recent_views 200 452.1 ms (6 DB calls)

We should profile this endpoint to see what is actually taking so long, but there are some possible avenues that we suspect:

  1. The 4 entity queries (:card, :dataset, :collection, and :table) are done sequentially, but could be done in parallel

  2. Removing runtime checks (costs: could return incorrect data in certain situations).

  3. Allow filtering via params (either count of items, or count per model, or just specify which models to grab)


  • profile endpoint and figure out what is expensive
@escherize escherize added .Task Not a part of any Epic, used by the Task Issue Template .Team/AdminWebapp Admin and Webapp team labels May 23, 2024
@escherize escherize self-assigned this May 23, 2024
@escherize
Copy link
Contributor Author

escherize commented May 23, 2024

Before any optimization, this is where we are on my m1 macbook pro:

on master:

Evaluation count : 4440 in 60 samples of 74 calls.
             Execution time mean : 15.107747 ms
    Execution time std-deviation : 1.086454 ms
   Execution time lower quantile : 13.752945 ms ( 2.5%)
   Execution time upper quantile : 17.359551 ms (97.5%)
                   Overhead used : 2.389407 ns

Found 1 outliers in 60 samples (1.6667 %)
	low-severe	 1 (1.6667 %)
 Variance from outliers : 53.4768 % Variance is severely inflated by outliers

Screenshot 2024-05-23 at 4 34 13 PM

Here's a flamegraph with recent-views highlighted. Seems like the time is indeed dominated by the big 4 (or 5) queries. I will see if parallelizing them makes a dent now.

@escherize
Copy link
Contributor Author

I suspect parallelizing the 4 big queries on the left would speed this up. On my small test data size, however, the overhead made it 10% slower.

@paoliniluis
Copy link
Contributor

I have a strong belief that a massive improvement might come if we go with #16909

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
.Backend .Performance .Task Not a part of any Epic, used by the Task Issue Template .Team/AdminWebapp Admin and Webapp team
Projects
None yet
Development

No branches or pull requests

2 participants