perf: Unbuffered cursors for large result sets

[ankush]

If you're reading 1000s of rows from MySQL, the default behaviour is to
read all of them in memory at once.

One of the use case for reading large rows is reporting where a lot of
data is read and then processed in Python. The read row is hoever not
used again but still consumes memory until entire function exits.

SSCursor (Server Side Cursor) allows fetching one row at a time.

Note: This is slower than fetching everything at once AND has risk of
connection loss. So, don't use this as a crutch. If possible rewrite
code so processing is done in SQL.

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
  1008    104.5 MiB    104.5 MiB           1    @profile
  1009                                          def test_run_memory_profile(self):
  1010    104.5 MiB      0.0 MiB           1            frappe.db.sql("select * from `tabGL Entry` limit 1")  # warmup
  1011    195.5 MiB      91 MiB       50001            for gl in frappe.db.sql("select * from `tabGL Entry` order by modified limit 50000", as_dict=True, as_iterator=True):
  1012    195.5 MiB      0.0 MiB       50000                    continue  # consume iterator
  1013
  1014    109.0 MiB    -86.5 MiB           1            pass # notice drop due to gc trigger

After:

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
  1013    104.5 MiB    104.5 MiB           1    @profile
  1014                                          def test_reads(self):
  1015    105.0 MiB      0.0 MiB           2            with frappe.db.unbuffered_cursor():
  1016    104.5 MiB      0.0 MiB           1                    frappe.db.sql("select * from `tabGL Entry` limit 1")  # warmup
  1017    105.0 MiB      0.5 MiB       50002                    for gl in frappe.db.sql(
  1018    104.5 MiB      0.0 MiB           1                            "select * from `tabGL Entry` order by modified limit 50000", as_dict=True, as_iterator=True
  1019                                                          ):
  1020    105.0 MiB      0.0 MiB       50000                            continue  # just consume the iterator
  1021    105.0 MiB      0.0 MiB           1                    pass

Extends #19810
Closes #18826

[codecov]

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (03b6d8a) 62.12% compared to head (ab96b60) 62.02%.
Report is 10 commits behind head on develop.

❗ Current head ab96b60 differs from pull request most recent head ff88fa0. Consider uploading reports for the commit ff88fa0 to get more accurate results

Additional details and impacted files

@@             Coverage Diff             @@
##           develop   #24365      +/-   ##
===========================================
- Coverage    62.12%   62.02%   -0.11%     
===========================================
  Files          786      786              
  Lines        74999    75139     +140     
  Branches      6422     6422              
===========================================
+ Hits         46596    46607      +11     
- Misses       24743    24872     +129     
  Partials      3660     3660              

Flag	Coverage Δ
server	70.93% <87.50%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.