perf: reduce DB calls made in `get_fetch_values` #17671

sagarvora · 2022-07-29T08:41:26Z

Changes Made

Get all values in one DB call, instead of a different DB call for each field to fetch.
Return frappe._dict instead of a regular dict
Don't make a DB call if value is falsy - assumption is that you're fetching from a non-single DocType (there's no reason to create a link field for single DocTypes)

Note: This function is used in ERPNext, but not Frappe.

Performance Improvement

Tested SalesInvoice.set_missing_lead_customer_details with line_profiler

Before

In [18]: %lprun -f get_fetch_values doc.set_missing_lead_customer_details()
Timer unit: 1e-06 s

Total time: 0.004495 s

After (~50% better)

In [20]: %lprun -f get_fetch_values doc.set_missing_lead_customer_details()
Timer unit: 1e-06 s

Total time: 0.002104 s

codecov · 2022-07-30T04:29:10Z

Codecov Report

Merging #17671 (3ef1c3e) into develop (9a50c30) will decrease coverage by 1.81%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           develop   #17671      +/-   ##
===========================================
- Coverage    59.62%   57.81%   -1.82%     
===========================================
  Files          759      759              
  Lines        68612    68620       +8     
  Branches      5971     5971              
===========================================
- Hits         40912    39672    -1240     
- Misses       24285    25533    +1248     
  Partials      3415     3415

Flag	Coverage Δ
server	`61.92% <100.00%> (-2.87%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

* perf: reduce DB calls made in `get_fetch_values` * fix: ensure return value is same as before * test: add test for `frappe.model.utils.get_fetch_values` (cherry picked from commit f9bfbfe)

* perf: reduce DB calls made in `get_fetch_values` * fix: ensure return value is same as before * test: add test for `frappe.model.utils.get_fetch_values` (cherry picked from commit f9bfbfe) Co-authored-by: Sagar Vora <sagar@resilient.tech>

* perf: reduce DB calls made in `get_fetch_values` * fix: ensure return value is same as before * test: add test for `frappe.model.utils.get_fetch_values` (cherry picked from commit f9bfbfe)

* perf: reduce DB calls made in `get_fetch_values` * fix: ensure return value is same as before * test: add test for `frappe.model.utils.get_fetch_values` (cherry picked from commit f9bfbfe) Co-authored-by: Sagar Vora <sagar@resilient.tech>

# [13.37.0](v13.36.3...v13.37.0) (2022-08-09) ### Bug Fixes * allow system managers to toggle email queue ([c004846](c004846)) * allow to import time field ([#17677](#17677)) ([#17690](#17690)) ([59f58c7](59f58c7)) * check permission before sending email ([bb73014](bb73014)) * Dropdown selection list in Reports should be translatable ([#17679](#17679)) ([8c0e5ab](8c0e5ab)) * **global_search:** Trigger rebuilding on Custom Field's property change ([#17705](#17705)) ([b4ff5b2](b4ff5b2)) * max_positive_value for Integer types ([#17712](#17712)) ([#17713](#17713)) ([e173e64](e173e64)) * merge conflict ([2f71fb1](2f71fb1)) * restored patch and remove unused import ([b2232f1](b2232f1)) * return promise ([#17646](#17646)) ([#17666](#17666)) ([d931e26](d931e26)) * Show Report & Dashboard View for File Doctype (backport [#17688](#17688)) ([#17695](#17695)) ([ad09e87](ad09e87)) * version bump regex in automated release ([#17725](#17725)) ([#17726](#17726)) ([a4f682a](a4f682a)) ### Features * Role based permission for Dashboard Chart ([#17634](#17634)) ([#17655](#17655)) ([c3f45e7](c3f45e7)) ### Performance Improvements * reduce DB calls in `frappe.client.get` (and other changes) (backport [#17665](#17665)) ([#17710](#17710)) ([5fe4924](5fe4924)) * reduce DB calls made in `get_fetch_values` ([#17671](#17671)) ([#17738](#17738)) ([fc58a87](fc58a87)) * specify reference doctype in filters ([#17703](#17703)) ([88a59d8](88a59d8))

# [14.1.0](v14.0.2...v14.1.0) (2022-08-11) ### Bug Fixes * add_user_permission with applicale_for arg set not updating in user permission ([804ce72](804ce72)) * assume parentfield to be set and valid ([e28674b](e28674b)) * check permlevel only if > 0 ([63f2cf5](63f2cf5)) * consistent fieldnames and keep port alongside server field ([c4f1619](c4f1619)) * **db:** Log mogrified queries ([ba1cdf8](ba1cdf8)) * dont init child doc ([94b760c](94b760c)) * dont pass doc to tooltip formatter on reportview ([d049c87](d049c87)) * german translations ([#17663](#17663)) ([#17771](#17771)) ([fd3eaa6](fd3eaa6)) * get correct doc when checking child table permission ([9b89dc6](9b89dc6)) * handle permlevel ([06ebd95](06ebd95)) * ignore built assets in translation ([53357e2](53357e2)) * ignore forms without docs in realtime events ([afc060f](afc060f)) * init only existing single doctypes ([75b6a33](75b6a33)) * limit allowed attributes for lazy images ([#17775](#17775)) ([1311564](1311564)) * **patch:** update multi step webform's Section Break fields to Page Break ([#17797](#17797)) ([#17798](#17798)) ([b3dc4c0](b3dc4c0)) * raise ImplicitCommitError instead of bare exception ([e2c6497](e2c6497)) * reverse logic for failing permission check ([7891674](7891674)) * **sanitize-html:** allow all data-* attrs ([9b94479](9b94479)) * send all messages on boot instead of scanning ([d61b7e8](d61b7e8)) * update append_to misleading message in email account ([#17718](#17718)) ([#17796](#17796)) ([cd86462](cd86462)) * **ux:** email domain doctype ([48c4a8d](48c4a8d)) * Webform Misc Fix (backport [#17642](#17642)) ([#17746](#17746)) ([86f9fbc](86f9fbc)) ### Features * JS extractor and custom JS parser ([fed4ae9](fed4ae9)) * **minor:** add connection link to email accounts in email domain ([821549d](821549d)) * **minor:** fetch attachment_limit from get_max_file_size api ([5190a49](5190a49)) * modified parser for frappe JS translate syntax ([0e87d21](0e87d21)) * Multilanguage sites: language selector only will show "enabled" languages and language default always will be System default ([#17745](#17745)) ([e5b8a47](e5b8a47)) * use AST to extract translation sources ([e02a73f](e02a73f)) ### Performance Improvements * reduce DB calls made in `get_fetch_values` ([#17671](#17671)) ([#17739](#17739)) ([dc19f69](dc19f69)) ### Reverts * Remove emoji from doctype ([c102124](c102124))

sagarvora requested a review from gavindsouza as a code owner July 29, 2022 08:41

sagarvora force-pushed the perf-get-fetch-values branch from bf9b71d to f1b1ebc Compare July 29, 2022 08:42

sagarvora added backport version-12-hotfix labels Jul 29, 2022

sagarvora force-pushed the perf-get-fetch-values branch 2 times, most recently from 6c6c983 to 1db7d2d Compare July 30, 2022 04:20

perf: reduce DB calls made in get_fetch_values

e5b4b06

sagarvora force-pushed the perf-get-fetch-values branch from 1db7d2d to e5b4b06 Compare August 2, 2022 03:32

sagarvora added the backport version-14-hotfix backport to version 14 label Aug 2, 2022

sagarvora marked this pull request as draft August 3, 2022 14:53

fix: ensure return value is same as before

93c9aee

sagarvora marked this pull request as ready for review August 5, 2022 05:48

Merge branch 'develop' into perf-get-fetch-values

8b5ce3e

sagarvora added the squash label Aug 5, 2022

ankush self-assigned this Aug 5, 2022

test: add test for frappe.model.utils.get_fetch_values

3ef1c3e

sagarvora requested a review from a team as a code owner August 5, 2022 06:06

ankush removed the request for review from gavindsouza August 5, 2022 06:09

ankush approved these changes Aug 5, 2022

View reviewed changes

ankush removed the backport version-12-hotfix label Aug 5, 2022

ankush merged commit f9bfbfe into frappe:develop Aug 5, 2022

ankush deleted the perf-get-fetch-values branch August 5, 2022 06:22

mergify bot mentioned this pull request Aug 5, 2022

perf: reduce DB calls made in get_fetch_values (backport #17671) #17738

Merged

mergify bot mentioned this pull request Aug 5, 2022

perf: reduce DB calls made in get_fetch_values (backport #17671) #17739

Merged

github-actions bot locked as resolved and limited conversation to collaborators Sep 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf: reduce DB calls made in `get_fetch_values` #17671

perf: reduce DB calls made in `get_fetch_values` #17671

sagarvora commented Jul 29, 2022 •

edited

codecov bot commented Jul 30, 2022 •

edited

perf: reduce DB calls made in get_fetch_values #17671

perf: reduce DB calls made in get_fetch_values #17671

Conversation

sagarvora commented Jul 29, 2022 • edited

Changes Made

Performance Improvement

Before

After (~50% better)

codecov bot commented Jul 30, 2022 • edited

Codecov Report

perf: reduce DB calls made in `get_fetch_values` #17671

perf: reduce DB calls made in `get_fetch_values` #17671

sagarvora commented Jul 29, 2022 •

edited

codecov bot commented Jul 30, 2022 •

edited