Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: cap max count on list views #25348

Merged
merged 6 commits into from
Mar 12, 2024
Merged

Conversation

ankush
Copy link
Member

@ankush ankush commented Mar 12, 2024

Problem: In InnoDB counting is essentially O(N) operation, it can be pretty fast on indexes so it doesn't feel like it most of the time but when filters don't use any index it usually means doing a full table scan.

get_count accounts for 4% of all requests on FC - #21742 . This isn't even considering outliers that were removed from data. Users with large DB often need to disable count on list views for site to stay usable.

This makes Count sad.
image

Fix: Adding a limit will stop the scan as soon as that many records are matched.

Implementation:

  • We only show accurate count up to 1000 values now
  • User can click on the count to get an accurate count if really required. Most users don't care about this at all!

count

Outcome:

  • If filters don't match more than 1k records then this PR makes no difference.
  • If filters match more than 1k records then all that extra cost is now not paid. So for site with millions of matching records get_count will finish much faster. The main benefit is user won't NEED to disable the count.

Count is happy now.

image

confuses query planner in some cases
@ankush ankush requested review from a team and akhilnarang and removed request for a team March 12, 2024 04:30
@github-actions github-actions bot added the add-test-cases Add test case to validate fix or enhancement label Mar 12, 2024
@ankush ankush added the defer backport Backports for some PR are deferred for a week or two to test them properly before releasing label Mar 12, 2024
@ankush
Copy link
Member Author

ankush commented Mar 12, 2024

Todo

  • tests
  • actual count formatting
  • child table (?)
  • avoid select star expansion
  • report view

@ankush ankush changed the title perf: show estimated max count on list views perf: cap max count on list views Mar 12, 2024
In InnoDB counting is essentially O(n) operation, it can be pretty fast
on indexes but when filters don't use any index it usually means doing a
full table scan.

Adding a limit will stop the scan as soon as that many records are
matched.
@ankush ankush enabled auto-merge March 12, 2024 09:38
@ankush ankush removed the add-test-cases Add test case to validate fix or enhancement label Mar 12, 2024
@ankush ankush disabled auto-merge March 12, 2024 09:42
@ankush ankush merged commit ee03bf7 into frappe:develop Mar 12, 2024
22 of 24 checks passed
@ankush ankush deleted the estimate_count branch March 12, 2024 10:06
@ankush ankush added backport version-15-hotfix Backport the PR to v15 and removed defer backport Backports for some PR are deferred for a week or two to test them properly before releasing labels Mar 12, 2024
ankush added a commit that referenced this pull request Mar 12, 2024
…-25348

perf: cap max count on list views (backport #25348)
ankush added a commit to ankush/frappe that referenced this pull request Mar 14, 2024
Order doesn't matter for count. This PR now actually makes frappe#25348 effective
ankush added a commit that referenced this pull request Mar 14, 2024
Order doesn't matter for count. This PR now actually makes #25348 effective
mergify bot pushed a commit that referenced this pull request Mar 14, 2024
Order doesn't matter for count. This PR now actually makes #25348 effective

(cherry picked from commit dbe55d4)
ankush added a commit that referenced this pull request Mar 14, 2024
Order doesn't matter for count. This PR now actually makes #25348 effective

(cherry picked from commit dbe55d4)

Co-authored-by: Ankush Menat <ankush@frappe.io>
@ankush ankush added the backport version-14-hotfix backport to version 14 label Mar 15, 2024
ankush added a commit that referenced this pull request Mar 15, 2024
* fix: dont add useless distinct clause

confuses query planner in some cases

(cherry picked from commit ae649aa)

# Conflicts:
#	frappe/desk/reportview.py

* feat: support countig till a limit

In InnoDB counting is essentially O(n) operation, it can be pretty fast
on indexes but when filters don't use any index it usually means doing a
full table scan.

Adding a limit will stop the scan as soon as that many records are
matched.

(cherry picked from commit a49fafb)

# Conflicts:
#	frappe/desk/reportview.py

* perf: show estimated count on list view

(cherry picked from commit 7e88c53)

* fix(UX): let user see actual count on click

(cherry picked from commit fdcff2d)

* fix: support child tables in count with limit

(cherry picked from commit 1fa7cc7)

* refactor: avoid duplicate render_count for report view

Only difference is element

(cherry picked from commit 698ef95)

* chore: conflicts

* fix: dont show tooltip if already limited (#25361)

---------

Co-authored-by: Ankush Menat <ankush@frappe.io>
@Tom-Finke
Copy link
Contributor

I got an error in one of my List Views and traced it to this change.
In Frappe Console do

import frappe.desk.reportview
frappe.local.form_dict = {"doctype": "User", "distinct": "true", "limit": 1001}
frappe.desk.reportview.get_count()

This gives DoesNotExistError: DocType tinct `tabUser` not found

@ankush
Copy link
Member Author

ankush commented Mar 18, 2024

#25511 should handle it.

frappe-pr-bot pushed a commit that referenced this pull request Mar 19, 2024
# [14.68.0](v14.67.1...v14.68.0) (2024-03-19)

### Bug Fixes

* allow exporting large reports that can't be rendered (backport [#25395](#25395)) ([#25404](#25404)) ([b3f4d5d](b3f4d5d))
* allow transitioning to long text (backport [#25419](#25419)) ([#25474](#25474)) ([60d8c1c](60d8c1c))
* avoid closing filter popover on any date picker interactions ([0784e61](0784e61))
* Avoid setting filter on now/today button ([83766c0](83766c0))
* child table rating fields ([#25433](#25433)) ([#25435](#25435)) ([57cf161](57cf161))
* consider all datepicker elements ([#25426](#25426)) ([ad79301](ad79301))
* debounce filter refresh ([7f65916](7f65916))
* don't allow setting an invalid rating ([#22633](#22633)) ([#25432](#25432)) ([502b907](502b907))
* dont add trailing decimal separator ([#25389](#25389)) ([#25390](#25390)) ([08e8d8c](08e8d8c))
* escape text types before setting disp area ([#25520](#25520)) ([#25522](#25522)) ([e446770](e446770))
* escape value in multiselect pill ([#25516](#25516)) ([#25517](#25517)) ([dbcf783](dbcf783))
* filters can use more than 140 chars ([d04bff9](d04bff9))
* handle distinct for fieldname (backport [#25511](#25511)) ([#25514](#25514)) ([3344df0](3344df0))
* handle parent rename in child workspace ([c3d9e1f](c3d9e1f))
* hide datepicker after picking date ([39796ea](39796ea))
* Log ipython commands (backport [#25364](#25364)) ([#25367](#25367)) ([997628c](997628c))
* log mariadb console usage ([6f4981c](6f4981c))
* show attachments on notifications too ([#25443](#25443)) ([#25448](#25448)) ([3251586](3251586))
* skip virtual doctype in tags ([b2ae437](b2ae437))
* Skip virtual doctype rename for dynamic links ([#25479](#25479)) ([#25480](#25480)) ([f368236](f368236))
* Skip virtual doctypes while renaming ([#25473](#25473)) ([#25477](#25477)) ([c423223](c423223))
* validate homepage paths (backport [#25409](#25409)) ([#25411](#25411)) ([ee1c8e3](ee1c8e3))

### Features

* connect to redis sentinel for redis cache (backport [#25398](#25398)) ([#25449](#25449)) ([6ce4036](6ce4036))
* debug stuck process by sending SIGUSR1 (backport [#25502](#25502)) ([#25526](#25526)) ([95d1224](95d1224))
* move bulk print operation to the background (backport [#25358](#25358)) ([#25396](#25396)) ([4508239](4508239))
* Store printed PDF attachments on communication (backport [#25439](#25439)) ([#25447](#25447)) ([9b7a02f](9b7a02f))

### Performance Improvements

* cap max count on list views (backport [#25348](#25348)) ([#25463](#25463)) ([3bc5d8a](3bc5d8a)), closes [#25361](#25361)
* compact prepared report files ([0aa3c8e](0aa3c8e))

### Reverts

* Revert "feat: move bulk print operation to the background (backport #25358) (#25396)" (#25400) ([490f031](490f031)), closes [#25358](#25358) [#25396](#25396) [#25400](#25400)
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 2, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
backport version-14-hotfix backport to version 14 backport version-15-hotfix Backport the PR to v15
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants