perf: improve responsiveness of payment reconciliation tool

[ruthra-kumar]

Issue

There are cases where customers have outstanding invoices in 10s of thousands and the reconciliation tool timesout on normal operations like fetching and allocating. This happens because, the Invoice Limit and Payment Limit are applied after pulling in all outstanding invoices and payments. This is inefficient.

Goal

To fix this permanently, a default limit of 50(which is still editable by customer) is applied and a search field is supplied for outstanding invoices and payments.

Search uses Indexed columns - voucher_no and against_voucher_no in Payment Ledger. This along with limit variable, makes the DB query fast. Fetch(Get Unreconciled Entries) becomes a constant time operation(O(n) -> O(50)) for default settings.

Changes

Limits are set to 50 as a default and are passed directly to the DB query. 2 new fields - Filter on Invoice and Filter on Payment, are added which pass its value to DB query. This way Vouchers are filtered at the query level.

before

In [13]: %time pr.get_unreconciled_entries()
CPU times: user 792 ms, sys: 29 ms, total: 821 ms
Wall time: 19.2 s

In [14]: len(pr.invoices)
Out[14]: 10000

In [15]: len(pr.payments)
Out[15]: 803

after

In [8]: %time pr.get_unreconciled_entries()
CPU times: user 107 ms, sys: 5.64 ms, total: 113 ms
Wall time: 643 ms

In [9]: len(pr.payments)
Out[9]: 50

In [10]: len(pr.invoices)
Out[10]: 50

after.mp4

[codecov]

Codecov Report

Merging #36650 (d01f0f2) into develop (8601e5b) will increase coverage by 0.12%.
Report is 44 commits behind head on develop.
The diff coverage is 83.67%.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop   #36650      +/-   ##
===========================================
+ Coverage    64.82%   64.94%   +0.12%     
===========================================
  Files          792      792              
  Lines        61798    61854      +56     
===========================================
+ Hits         40063    40174     +111     
+ Misses       21735    21680      -55     

Files Changed	Coverage Δ
...ype/pos_invoice_merge_log/pos_invoice_merge_log.py	79.80% <ø> (ø)
erpnext/controllers/accounts_controller.py	86.10% <ø> (-0.01%)	⬇️
...e/payment_reconciliation/payment_reconciliation.py	87.62% <72.72%> (-0.59%)	⬇️
erpnext/accounts/utils.py	73.99% <77.77%> (+0.01%)	⬆️
...xt/accounts/doctype/payment_entry/payment_entry.py	83.50% <87.50%> (-0.22%)	⬇️
...rpnext/accounts/doctype/pos_invoice/pos_invoice.py	73.37% <100.00%> (+0.09%)	⬆️
...serial_and_batch_bundle/serial_and_batch_bundle.py	71.15% <100.00%> (-0.06%)	⬇️

... and 11 files with indirect coverage changes