Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: cap max count on list views (backport #25348) #25463

Merged
merged 8 commits into from
Mar 15, 2024

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented Mar 15, 2024

Problem: In InnoDB counting is essentially O(N) operation, it can be pretty fast on indexes so it doesn't feel like it most of the time but when filters don't use any index it usually means doing a full table scan.

get_count accounts for 4% of all requests on FC - #21742 . This isn't even considering outliers that were removed from data. Users with large DB often need to disable count on list views for site to stay usable.

This makes Count sad.
image

Fix: Adding a limit will stop the scan as soon as that many records are matched.

Implementation:

  • We only show accurate count up to 1000 values now
  • User can click on the count to get an accurate count if really required. Most users don't care about this at all!

count

Outcome:

  • If filters don't match more than 1k records then this PR makes no difference.
  • If filters match more than 1k records then all that extra cost is now not paid. So for site with millions of matching records get_count will finish much faster. The main benefit is user won't NEED to disable the count.

Count is happy now.

image


This is an automatic backport of pull request #25348 done by [Mergify](https://mergify.com).

confuses query planner in some cases

(cherry picked from commit ae649aa)

# Conflicts:
#	frappe/desk/reportview.py
In InnoDB counting is essentially O(n) operation, it can be pretty fast
on indexes but when filters don't use any index it usually means doing a
full table scan.

Adding a limit will stop the scan as soon as that many records are
matched.

(cherry picked from commit a49fafb)

# Conflicts:
#	frappe/desk/reportview.py
Only difference is element

(cherry picked from commit 698ef95)
@mergify mergify bot requested a review from a team March 15, 2024 08:52
@mergify mergify bot added the conflicts label Mar 15, 2024
@mergify mergify bot requested review from ankush and removed request for a team March 15, 2024 08:52
@mergify mergify bot assigned ankush Mar 15, 2024
Copy link
Contributor Author

mergify bot commented Mar 15, 2024

Cherry-pick of ae649aa has failed:

On branch mergify/bp/version-14-hotfix/pr-25348
Your branch is up to date with 'origin/version-14-hotfix'.

You are currently cherry-picking commit ae649aadf0.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   frappe/desk/reportview.py

no changes added to commit (use "git add" and/or "git commit -a")

Cherry-pick of a49fafb has failed:

On branch mergify/bp/version-14-hotfix/pr-25348
Your branch is ahead of 'origin/version-14-hotfix' by 1 commit.
  (use "git push" to publish your local commits)

You are currently cherry-picking commit a49fafbf8e.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   frappe/public/js/frappe/db.js
	modified:   frappe/public/js/frappe/list/list_view.js
	modified:   frappe/tests/test_db_query.py

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   frappe/desk/reportview.py

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

@ankush ankush enabled auto-merge (squash) March 15, 2024 09:10
@ankush ankush merged commit 3bc5d8a into version-14-hotfix Mar 15, 2024
8 of 11 checks passed
@ankush ankush deleted the mergify/bp/version-14-hotfix/pr-25348 branch March 15, 2024 09:23
frappe-pr-bot pushed a commit that referenced this pull request Mar 19, 2024
# [14.68.0](v14.67.1...v14.68.0) (2024-03-19)

### Bug Fixes

* allow exporting large reports that can't be rendered (backport [#25395](#25395)) ([#25404](#25404)) ([b3f4d5d](b3f4d5d))
* allow transitioning to long text (backport [#25419](#25419)) ([#25474](#25474)) ([60d8c1c](60d8c1c))
* avoid closing filter popover on any date picker interactions ([0784e61](0784e61))
* Avoid setting filter on now/today button ([83766c0](83766c0))
* child table rating fields ([#25433](#25433)) ([#25435](#25435)) ([57cf161](57cf161))
* consider all datepicker elements ([#25426](#25426)) ([ad79301](ad79301))
* debounce filter refresh ([7f65916](7f65916))
* don't allow setting an invalid rating ([#22633](#22633)) ([#25432](#25432)) ([502b907](502b907))
* dont add trailing decimal separator ([#25389](#25389)) ([#25390](#25390)) ([08e8d8c](08e8d8c))
* escape text types before setting disp area ([#25520](#25520)) ([#25522](#25522)) ([e446770](e446770))
* escape value in multiselect pill ([#25516](#25516)) ([#25517](#25517)) ([dbcf783](dbcf783))
* filters can use more than 140 chars ([d04bff9](d04bff9))
* handle distinct for fieldname (backport [#25511](#25511)) ([#25514](#25514)) ([3344df0](3344df0))
* handle parent rename in child workspace ([c3d9e1f](c3d9e1f))
* hide datepicker after picking date ([39796ea](39796ea))
* Log ipython commands (backport [#25364](#25364)) ([#25367](#25367)) ([997628c](997628c))
* log mariadb console usage ([6f4981c](6f4981c))
* show attachments on notifications too ([#25443](#25443)) ([#25448](#25448)) ([3251586](3251586))
* skip virtual doctype in tags ([b2ae437](b2ae437))
* Skip virtual doctype rename for dynamic links ([#25479](#25479)) ([#25480](#25480)) ([f368236](f368236))
* Skip virtual doctypes while renaming ([#25473](#25473)) ([#25477](#25477)) ([c423223](c423223))
* validate homepage paths (backport [#25409](#25409)) ([#25411](#25411)) ([ee1c8e3](ee1c8e3))

### Features

* connect to redis sentinel for redis cache (backport [#25398](#25398)) ([#25449](#25449)) ([6ce4036](6ce4036))
* debug stuck process by sending SIGUSR1 (backport [#25502](#25502)) ([#25526](#25526)) ([95d1224](95d1224))
* move bulk print operation to the background (backport [#25358](#25358)) ([#25396](#25396)) ([4508239](4508239))
* Store printed PDF attachments on communication (backport [#25439](#25439)) ([#25447](#25447)) ([9b7a02f](9b7a02f))

### Performance Improvements

* cap max count on list views (backport [#25348](#25348)) ([#25463](#25463)) ([3bc5d8a](3bc5d8a)), closes [#25361](#25361)
* compact prepared report files ([0aa3c8e](0aa3c8e))

### Reverts

* Revert "feat: move bulk print operation to the background (backport #25358) (#25396)" (#25400) ([490f031](490f031)), closes [#25358](#25358) [#25396](#25396) [#25400](#25400)
@frappe-pr-bot
Copy link
Collaborator

🎉 This PR is included in version 14.68.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

akhilnarang added a commit that referenced this pull request Mar 21, 2024
Got removed in #25463

Signed-off-by: Akhil Narang <me@akhilnarang.dev>
frappe-pr-bot pushed a commit that referenced this pull request Mar 21, 2024
## [14.68.1](v14.68.0...v14.68.1) (2024-03-21)

### Bug Fixes

* add missing import ([beb2f22](beb2f22)), closes [#25463](#25463)
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 10, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants