-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
perf: cap max count on list views #25348
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
confuses query planner in some cases
Todo
|
In InnoDB counting is essentially O(n) operation, it can be pretty fast on indexes but when filters don't use any index it usually means doing a full table scan. Adding a limit will stop the scan as soon as that many records are matched.
Only difference is element
ankush
added a commit
that referenced
this pull request
Mar 12, 2024
…-25348 perf: cap max count on list views (backport #25348)
ankush
added a commit
to ankush/frappe
that referenced
this pull request
Mar 14, 2024
Order doesn't matter for count. This PR now actually makes frappe#25348 effective
ankush
added a commit
that referenced
this pull request
Mar 14, 2024
Order doesn't matter for count. This PR now actually makes #25348 effective
ankush
added a commit
that referenced
this pull request
Mar 14, 2024
ankush
added a commit
that referenced
this pull request
Mar 15, 2024
* fix: dont add useless distinct clause confuses query planner in some cases (cherry picked from commit ae649aa) # Conflicts: # frappe/desk/reportview.py * feat: support countig till a limit In InnoDB counting is essentially O(n) operation, it can be pretty fast on indexes but when filters don't use any index it usually means doing a full table scan. Adding a limit will stop the scan as soon as that many records are matched. (cherry picked from commit a49fafb) # Conflicts: # frappe/desk/reportview.py * perf: show estimated count on list view (cherry picked from commit 7e88c53) * fix(UX): let user see actual count on click (cherry picked from commit fdcff2d) * fix: support child tables in count with limit (cherry picked from commit 1fa7cc7) * refactor: avoid duplicate render_count for report view Only difference is element (cherry picked from commit 698ef95) * chore: conflicts * fix: dont show tooltip if already limited (#25361) --------- Co-authored-by: Ankush Menat <ankush@frappe.io>
I got an error in one of my List Views and traced it to this change.
This gives |
#25511 should handle it. |
frappe-pr-bot
pushed a commit
that referenced
this pull request
Mar 19, 2024
# [14.68.0](v14.67.1...v14.68.0) (2024-03-19) ### Bug Fixes * allow exporting large reports that can't be rendered (backport [#25395](#25395)) ([#25404](#25404)) ([b3f4d5d](b3f4d5d)) * allow transitioning to long text (backport [#25419](#25419)) ([#25474](#25474)) ([60d8c1c](60d8c1c)) * avoid closing filter popover on any date picker interactions ([0784e61](0784e61)) * Avoid setting filter on now/today button ([83766c0](83766c0)) * child table rating fields ([#25433](#25433)) ([#25435](#25435)) ([57cf161](57cf161)) * consider all datepicker elements ([#25426](#25426)) ([ad79301](ad79301)) * debounce filter refresh ([7f65916](7f65916)) * don't allow setting an invalid rating ([#22633](#22633)) ([#25432](#25432)) ([502b907](502b907)) * dont add trailing decimal separator ([#25389](#25389)) ([#25390](#25390)) ([08e8d8c](08e8d8c)) * escape text types before setting disp area ([#25520](#25520)) ([#25522](#25522)) ([e446770](e446770)) * escape value in multiselect pill ([#25516](#25516)) ([#25517](#25517)) ([dbcf783](dbcf783)) * filters can use more than 140 chars ([d04bff9](d04bff9)) * handle distinct for fieldname (backport [#25511](#25511)) ([#25514](#25514)) ([3344df0](3344df0)) * handle parent rename in child workspace ([c3d9e1f](c3d9e1f)) * hide datepicker after picking date ([39796ea](39796ea)) * Log ipython commands (backport [#25364](#25364)) ([#25367](#25367)) ([997628c](997628c)) * log mariadb console usage ([6f4981c](6f4981c)) * show attachments on notifications too ([#25443](#25443)) ([#25448](#25448)) ([3251586](3251586)) * skip virtual doctype in tags ([b2ae437](b2ae437)) * Skip virtual doctype rename for dynamic links ([#25479](#25479)) ([#25480](#25480)) ([f368236](f368236)) * Skip virtual doctypes while renaming ([#25473](#25473)) ([#25477](#25477)) ([c423223](c423223)) * validate homepage paths (backport [#25409](#25409)) ([#25411](#25411)) ([ee1c8e3](ee1c8e3)) ### Features * connect to redis sentinel for redis cache (backport [#25398](#25398)) ([#25449](#25449)) ([6ce4036](6ce4036)) * debug stuck process by sending SIGUSR1 (backport [#25502](#25502)) ([#25526](#25526)) ([95d1224](95d1224)) * move bulk print operation to the background (backport [#25358](#25358)) ([#25396](#25396)) ([4508239](4508239)) * Store printed PDF attachments on communication (backport [#25439](#25439)) ([#25447](#25447)) ([9b7a02f](9b7a02f)) ### Performance Improvements * cap max count on list views (backport [#25348](#25348)) ([#25463](#25463)) ([3bc5d8a](3bc5d8a)), closes [#25361](#25361) * compact prepared report files ([0aa3c8e](0aa3c8e)) ### Reverts * Revert "feat: move bulk print operation to the background (backport #25358) (#25396)" (#25400) ([490f031](490f031)), closes [#25358](#25358) [#25396](#25396) [#25400](#25400)
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Labels
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem: In InnoDB counting is essentially O(N) operation, it can be pretty fast on indexes so it doesn't feel like it most of the time but when filters don't use any index it usually means doing a full table scan.
get_count
accounts for 4% of all requests on FC - #21742 . This isn't even considering outliers that were removed from data. Users with large DB often need to disable count on list views for site to stay usable.This makes Count sad.
![image](https://private-user-images.githubusercontent.com/9079960/311942177-a6b46223-66ee-49ee-bf90-0a6f092cdf2a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjI0MTM2MzUsIm5iZiI6MTcyMjQxMzMzNSwicGF0aCI6Ii85MDc5OTYwLzMxMTk0MjE3Ny1hNmI0NjIyMy02NmVlLTQ5ZWUtYmY5MC0wYTZmMDkyY2RmMmEucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDczMSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA3MzFUMDgwODU1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NWY4NzMyZGZhNTZhYTA0N2Q5ZDUyZTUwMjZhNTQwOTg1NjIwNzRlYzNjOTk0OTNmNzhjYzJkNGY5ZDdiNTU5MCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.DlgW96qrd-K3v4bMg8PXv0jPnpQ9nrHdJ9Zltu6dImk)
Fix: Adding a limit will stop the scan as soon as that many records are matched.
Implementation:
Outcome:
Count is happy now.