-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
perf: cap max count on list views (backport #25348) #25463
Merged
ankush
merged 8 commits into
version-14-hotfix
from
mergify/bp/version-14-hotfix/pr-25348
Mar 15, 2024
Merged
perf: cap max count on list views (backport #25348) #25463
ankush
merged 8 commits into
version-14-hotfix
from
mergify/bp/version-14-hotfix/pr-25348
Mar 15, 2024
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
confuses query planner in some cases (cherry picked from commit ae649aa) # Conflicts: # frappe/desk/reportview.py
In InnoDB counting is essentially O(n) operation, it can be pretty fast on indexes but when filters don't use any index it usually means doing a full table scan. Adding a limit will stop the scan as soon as that many records are matched. (cherry picked from commit a49fafb) # Conflicts: # frappe/desk/reportview.py
(cherry picked from commit 7e88c53)
(cherry picked from commit fdcff2d)
(cherry picked from commit 1fa7cc7)
Only difference is element (cherry picked from commit 698ef95)
Cherry-pick of ae649aa has failed:
Cherry-pick of a49fafb has failed:
To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally |
frappe-pr-bot
pushed a commit
that referenced
this pull request
Mar 19, 2024
# [14.68.0](v14.67.1...v14.68.0) (2024-03-19) ### Bug Fixes * allow exporting large reports that can't be rendered (backport [#25395](#25395)) ([#25404](#25404)) ([b3f4d5d](b3f4d5d)) * allow transitioning to long text (backport [#25419](#25419)) ([#25474](#25474)) ([60d8c1c](60d8c1c)) * avoid closing filter popover on any date picker interactions ([0784e61](0784e61)) * Avoid setting filter on now/today button ([83766c0](83766c0)) * child table rating fields ([#25433](#25433)) ([#25435](#25435)) ([57cf161](57cf161)) * consider all datepicker elements ([#25426](#25426)) ([ad79301](ad79301)) * debounce filter refresh ([7f65916](7f65916)) * don't allow setting an invalid rating ([#22633](#22633)) ([#25432](#25432)) ([502b907](502b907)) * dont add trailing decimal separator ([#25389](#25389)) ([#25390](#25390)) ([08e8d8c](08e8d8c)) * escape text types before setting disp area ([#25520](#25520)) ([#25522](#25522)) ([e446770](e446770)) * escape value in multiselect pill ([#25516](#25516)) ([#25517](#25517)) ([dbcf783](dbcf783)) * filters can use more than 140 chars ([d04bff9](d04bff9)) * handle distinct for fieldname (backport [#25511](#25511)) ([#25514](#25514)) ([3344df0](3344df0)) * handle parent rename in child workspace ([c3d9e1f](c3d9e1f)) * hide datepicker after picking date ([39796ea](39796ea)) * Log ipython commands (backport [#25364](#25364)) ([#25367](#25367)) ([997628c](997628c)) * log mariadb console usage ([6f4981c](6f4981c)) * show attachments on notifications too ([#25443](#25443)) ([#25448](#25448)) ([3251586](3251586)) * skip virtual doctype in tags ([b2ae437](b2ae437)) * Skip virtual doctype rename for dynamic links ([#25479](#25479)) ([#25480](#25480)) ([f368236](f368236)) * Skip virtual doctypes while renaming ([#25473](#25473)) ([#25477](#25477)) ([c423223](c423223)) * validate homepage paths (backport [#25409](#25409)) ([#25411](#25411)) ([ee1c8e3](ee1c8e3)) ### Features * connect to redis sentinel for redis cache (backport [#25398](#25398)) ([#25449](#25449)) ([6ce4036](6ce4036)) * debug stuck process by sending SIGUSR1 (backport [#25502](#25502)) ([#25526](#25526)) ([95d1224](95d1224)) * move bulk print operation to the background (backport [#25358](#25358)) ([#25396](#25396)) ([4508239](4508239)) * Store printed PDF attachments on communication (backport [#25439](#25439)) ([#25447](#25447)) ([9b7a02f](9b7a02f)) ### Performance Improvements * cap max count on list views (backport [#25348](#25348)) ([#25463](#25463)) ([3bc5d8a](3bc5d8a)), closes [#25361](#25361) * compact prepared report files ([0aa3c8e](0aa3c8e)) ### Reverts * Revert "feat: move bulk print operation to the background (backport #25358) (#25396)" (#25400) ([490f031](490f031)), closes [#25358](#25358) [#25396](#25396) [#25400](#25400)
🎉 This PR is included in version 14.68.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
akhilnarang
added a commit
that referenced
this pull request
Mar 21, 2024
Got removed in #25463 Signed-off-by: Akhil Narang <me@akhilnarang.dev>
frappe-pr-bot
pushed a commit
that referenced
this pull request
Mar 21, 2024
## [14.68.1](v14.68.0...v14.68.1) (2024-03-21) ### Bug Fixes * add missing import ([beb2f22](beb2f22)), closes [#25463](#25463)
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem: In InnoDB counting is essentially O(N) operation, it can be pretty fast on indexes so it doesn't feel like it most of the time but when filters don't use any index it usually means doing a full table scan.
get_count
accounts for 4% of all requests on FC - #21742 . This isn't even considering outliers that were removed from data. Users with large DB often need to disable count on list views for site to stay usable.This makes Count sad.
![image](https://private-user-images.githubusercontent.com/9079960/311942177-a6b46223-66ee-49ee-bf90-0a6f092cdf2a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk1Mjk4MTMsIm5iZiI6MTcxOTUyOTUxMywicGF0aCI6Ii85MDc5OTYwLzMxMTk0MjE3Ny1hNmI0NjIyMy02NmVlLTQ5ZWUtYmY5MC0wYTZmMDkyY2RmMmEucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYyNyUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MjdUMjMwNTEzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NTRlYzY1NGU4MDFkYjE0YjYzMDQzNzhhOTc2ZDg5ZGMyMGVkZTkwOWU5ZmY1NTg0MmMzYTc1MjJkOWI1YjIyNSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.QY1f77ukobh7XVS2VgFls0d7eEOZFRq02Vvxzbb_eLc)
Fix: Adding a limit will stop the scan as soon as that many records are matched.
Implementation:
Outcome:
Count is happy now.
This is an automatic backport of pull request #25348 done by [Mergify](https://mergify.com).