Skip to content

Conversation

@fw-bot
Copy link
Contributor

@fw-bot fw-bot commented Apr 24, 2025

Description

I am experiencing slow processing times for incoming stock from vendors when the historical data contains a large number of records.

Currently, my company has around 20 million stock.valuation.layer records. As a result, any transaction involving incoming stock takes approximately 5–21 seconds to complete, even for a small number of products (around 1–5 products).

Through profiling, I have identified that the root cause is the compute function _compute_value_svl. This computed field is always triggered during incoming stock transactions when calculating the average cost or fixing rounding errors.

Proposal

To improve performance, I propose implementing a Parallel Index-Only Scan. This method is effective for grouping large datasets while retrieving only fixed columns, optimizing query execution.

Benchmark

Before After Speedup
1.1 s 97.8 ms 11x

Before

Finalize GroupAggregate  (cost=1000.56..397852.39 rows=538 width=80) (actual time=1040.355..1114.694 rows=1 loops=1)
  Group Key: product_id
  Buffers: shared hit=227534 read=330906 dirtied=83 written=683
  ->  Gather  (cost=1000.56..397825.49 rows=1076 width=80) (actual time=1039.900..1114.679 rows=3 loops=1)
        Workers Planned: 2
        Workers Launched: 2
        Buffers: shared hit=227534 read=330906 dirtied=83 written=683
        ->  Partial GroupAggregate  (cost=0.56..396717.89 rows=538 width=80) (actual time=1036.758..1036.759 rows=1 loops=3)
              Group Key: product_id
              Buffers: shared hit=227534 read=330906 dirtied=83 written=683
              ->  Parallel Index Scan using stock_valuation_layer_index on stock_valuation_layer  
                    (cost=0.56..392927.16 rows=302613 width=20) (actual time=0.608..975.431 rows=247487 loops=3)
                    Index Cond: ((product_id = 140) AND (company_id = 1))
                    Buffers: shared hit=227534 read=330906 dirtied=83 written=683
Planning:
  Buffers: shared hit=9 dirtied=1
Planning Time: 0.139 ms
Execution Time: 1114.732 ms

After

Finalize GroupAggregate  (cost=1000.56..21876.83 rows=538 width=80) (actual time=91.888..97.719 rows=1 loops=1)
  Group Key: product_id
  Buffers: shared hit=12889 read=7121
  ->  Gather  (cost=1000.56..21849.93 rows=1076 width=80) (actual time=91.636..97.704 rows=3 loops=1)
        Workers Planned: 2
        Workers Launched: 2
        Buffers: shared hit=12889 read=7121
        ->  Partial GroupAggregate  (cost=0.56..20742.33 rows=538 width=80) (actual time=87.579..87.580 rows=1 loops=3)
              Group Key: product_id
              Buffers: shared hit=12889 read=7121
              ->  Parallel Index Only Scan using idx_svl_company_product on stock_valuation_layer  
                    (cost=0.56..16951.60 rows=302613 width=20) (actual time=0.044..46.642 rows=247487 loops=3)
                    Index Cond: ((company_id = 1) AND (product_id = 140))
                    Heap Fetches: 9188
                    Buffers: shared hit=12889 read=7121
Planning Time: 0.107 ms
Execution Time: 97.751 ms

I confirm I have signed the CLA and read the PR guidelines at www.odoo.com/submit-pr

Forward-Port-Of: #206587
Forward-Port-Of: #203038

@robodoo
Copy link
Contributor

robodoo commented Apr 24, 2025

Pull request status dashboard

@robodoo robodoo added forwardport This PR was created by @fw-bot conflict There was an error while creating this forward-port PR labels Apr 24, 2025
@fw-bot
Copy link
Contributor Author

fw-bot commented Apr 24, 2025

@paimonchan @Whenrow cherrypicking of pull request #203038 failed.

stdout:

Auto-merging addons/stock_account/models/stock_valuation_layer.py
CONFLICT (content): Merge conflict in addons/stock_account/models/stock_valuation_layer.py

Either perform the forward-port manually (and push to this branch, proceeding as usual) or close this PR (maybe?).

In the former case, you may want to edit this PR message as well.

⚠️ after resolving this conflict, you will need to merge it via @robodoo.

More info at https://github.com/odoo/odoo/wiki/Mergebot#forward-port

@C3POdoo C3POdoo added the RD research & development, internal work label Apr 24, 2025
@Whenrow Whenrow force-pushed the saas-18.1-16.0-perf-optimize-compute-valuation-431572-fw branch from 8417198 to b82c554 Compare April 25, 2025 06:41
@Whenrow Whenrow force-pushed the saas-18.1-16.0-perf-optimize-compute-valuation-431572-fw branch from b82c554 to 2e0f46a Compare April 25, 2025 06:43
@Whenrow
Copy link
Contributor

Whenrow commented Apr 25, 2025

robodoo r+

@C3POdoo C3POdoo requested a review from a team April 25, 2025 06:44
@robodoo
Copy link
Contributor

robodoo commented Apr 25, 2025

@paimonchan @Whenrow 'ci/runbot' failed on this reviewed PR.

robodoo pushed a commit that referenced this pull request Apr 25, 2025
Part-of: #207311
Signed-off-by: William Henrotin (whe) <whe@odoo.com>
robodoo pushed a commit that referenced this pull request Apr 25, 2025
closes #207311

Signed-off-by: William Henrotin (whe) <whe@odoo.com>
@robodoo robodoo closed this Apr 25, 2025
@fw-bot fw-bot deleted the saas-18.1-16.0-perf-optimize-compute-valuation-431572-fw branch May 2, 2025 17:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

conflict There was an error while creating this forward-port PR forwardport This PR was created by @fw-bot RD research & development, internal work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants