[PERF] How we can help you to improve the performance of Odoo ?

[fmdl]

Hi @odony @rco-odoo @nim-odoo
Last month I have make some differente issues/PR about the performance of Odoo. I have the impression that it remains a dead letter.

I give you some improvements, some cases where the performance is not at hand, I'm ready to help you, but it seems that this is not the priority. This is a shame because I have customers who are not happy with the performance, while overall Odoo is very successful.

I apologize in advance for giving you work, but we (and other developers) we must have solutions to handle this is case (order with 5,000 lines, receipt with 20,000 serial numbers, product with 5,000 variants ...).

Coud you work on a roadmap about perf issue.

Resumé of différente perf issue:

• product.product_tmpl_id.uom_id is more fast than product.uom_id (see : [PERF][11.0] access to related field with very bad performance #24119 and [PERF] stock : fast compute when picking with 2000 product with unique serial number #23784)
• unlink clear all cache (to much), 2 secondes to action_done one serial product in picking of 5000 product : [PERF] quant: direct SQL to prevent clean of all cache : 350% more faster ! #25793 (total time 3 hours ...).
• create clean less cache than unlink, but clean to much cache, issue when create stock.move.line with product with serial number (see: [FIX][Performance] product: recompute fields of all new variants in one time #23108)
• read_group : remove the name_get() during computing [FIX] base: allow to no compute name_get during read_group #23755
• add read_group for each UI computed field for products, sale, purchase, picking, ...: [11.0][Poor Performance] when open a product with 5000 variantes #22783
• improve read_group to pass more complex SQL query to compute UI fields more quickly
• improve mapped function are also slowly sum(record.mapped('lines')), with 5000 lines take long long time. see [FIX] base: fast mapped func + 1800 % #30975
• create ui_field (see bellow)

Reflexion about computed field
We have an issue with computed field, when the time to compute is not some millisecondes. (like picking state)
Store=True

• good performance to show a record in web UI
• bad performance in computation, before all computed field are computed at each write and not at the end of calculation.

Store=False

• good performance in computation, fields are only recomputed when needed (after a clear with depends)
• bad performance for UI

The best thing for next version, is to have good perf in both case :-).
When Store=True proposition:

• recompute when needed and only at the end of action (ex : action assign). Attention before any manual SQL query, don't forget to recompute and save fields.

When Store=False proposition:

• keep a copy of the value in DB, this DB value is used only in readonly mode in the UI. The value can be recompute by a cron.
• asynchrone the showing of value in UI

Maybe create a type field use only for UI with a specifique compute system (on other thread, asynchrone UI...). In standard this field are not used, but can be used by developper/integrator for specific application.

When the fields are not linked, we could compute in multithread.

Thanks for your work.

Best regards

cc @fpodoo

[Yenthe666]

@nseinlet as it is about performance I think you might like this topic too.

[odony]

@fmdl Performance issues are very tricky. Each case requires a careful analysis of the root cause, and the evaluation of possible options.

We are even more careful with performance improvements than with real bugs, because they are often risky changes that can compromise the stability of our product. And when it happens it impacts all the other users who did not even suffer from the performance problem in the first place. That is a bad risk/benefit ratio.

There is also a limit to what can should be done in the standard version of Odoo in terms of performance. Some extreme cases are much better solved with customer-specific patches rather than more invasive and dangerous patches in the framework.

The data mapping also needs to be valid in the first place. For example, products in Odoo are not designed to handle thousands of variants. Variants are meant for products with a few attributes and a few attributes values, such as color and size. You can of course misuse them to produce a combination of attributes with thousands or millions of variations. When you do that, you're choosing the wrong model. The solution is to change the way this use case is mapped, rather than trying to gracefully handle a volume of data that the feature is not designed for. A business solution is more appropriate than a technical one.

Long story short, for each of the topic you mention, you and us both need to evaluate if something should be done in standard, and then decide if we can do it in stable versions, or if it requires deeper changes, better suited for master. Not an easy task.

PS: nothing prevents you from testing patches on real deployments before they are merged in the official version. This will even allow you to report back on any results and side-effects you noticed.

[odony]

To illustrate this, your PR #23755 that we previously discussed has been improved and the patch was merged yesterday in master: fd9de4d

[odony]

By the way, we really appreciate that you are working on these difficult performance issues and that you are taking the time to propose POC patches and creative ideas. Thanks a lot for that, it's awesome! :-)

Just keep in mind that these are difficult issues to fix, patches are hard to verify thoroughly (side-effects!), so please bear with us and be patient, for all the reasons explained before.

Our framework team is continuously working on performance improvements, do not hesitate to keep discussing those issues and suggestions with @rco-odoo and @xmo-odoo, for instance.

[fmdl]

Thank for your reply. I do not ask to solve everything quickly, but more to have a vision on your part of how you plan to improve this in the long term.
One last note, the understanding of the management of the cache on Odoo is rather difficult / complex (shared between model.py, fields.py and api.py), could not one group together the functioning of the cache (creation, reading , erase, when it is clean, with_prefectch, prefetch_fields = False, ...) in a specific documentation ?

[blaggacao]

For reference (create ina for loop): #25912 (Issue) & #25978 (PR)

[felipepquim]

I have been using odoo since january in my company (>20 users) and though I am overall satisfied with the Odoo, the system is really slow in several instances, and the POS systems has given many problems. Hoping that this 2 issues are improved.

[moylop260]

What about uses multi records for convert_to_read or name_get instead of uses one record each time?

• https://github.com/odoo/odoo/blame/b623695ee76b22be2b50d5eeb51e8a22bb813660/odoo/models.py#L2601

If you have a view with 10 m2o fields for 80 records the name_get method is called 800 times instead of just 10 times (with all 80 records together) since that the methods support multiple records to process.

[rvalyi]

At Akretion we are dealing with Odoo ecommerce with thousands of variantes since nearly a decade so we know the solution. A few years ago we contributes the fields.Serialized fields into the core of Odoo. Basically it allows to do NoSQL with Odoo and store your variant information in JSON.

If it is for e-commerce you could give a try to Shopinvader or extract a solution from the Shopinvader codebase. Here is how we encore the product variants in NoSQL:
https://github.com/akretion/odoo-shopinvader/blob/27a15ec3c1f63b42e151137189c2ac5748377f18/shopinvader/models/shopinvader_variant.py#L43
Then we are indexing the json records in Algolia and Shopinvader consumes them directly from Algolia, which gives us unparalleled performance and total security of the ERP (no SQL injection in your ERP via a rating widget for instance).

Orders with many lines is also another tricky point with the chain of computed fields. Several guys are able to fix this, but eventually you can contact Acsone as they had to fix this kind of thing on the new API.