Read_group: A New Hope

[ryv-odoo]

Rationale

The method read_group() was designed to be used by the web client to efficiently compute aggregations grouped by one or more fields. However, more and more developers have been using it from the backend to make computations more efficient (avoid doing the aggregation in Python). Unfortunately, the API was designed for the web client, which adds a lot of boilerplate when used in the Python (list of dict with misleading key names).

Method _read_group() was created to improve the performance of read_group() for backend use (4ef0c00), but didn't change the API and based the implementation on read_group() itself.

This rewrites _read_group() from scratch with a new API to make it easier to use from the backend. Also, split the method to make it easy to override and add custom behavior.

New API of _read_group()

The purpose of the method is unchanged: get field aggregations specified by aggregates grouped by the given groupby fields where records are filtered by the domain.

Parameters

• Rename fields into aggregates; parameter aggregates is a list of explicit aggregate specifications (no implicit aggregator coming from group_operator).
• Parameter groupby becomes the second parameter (just before aggregates). It is much more readable for the returning value. Also the groupby specification for date/datetime fields should be always explicit (no default 'month').
• New parameter having, which is a domain-like list to filter the result depending of aggregate values. It allows to simplify some code and avoid to fetch unnecessary data from the database.
• Parameter orderby is renamed into order, to be consistent with method search(). Order terms should be an explicit aggregate specification or groupby specification (+ ASC/DESC).
• Remove parameter lazy, which was only used for the webclient and doesn't have any sense to be used in backend.

Returned value

The returned value is a list of tuples containing, in order, the group values and aggregate values (flatten). For instance, if you group by ['foo', 'bar'] and aggregate ['baz:sum', 'quux:max'], the result will contain tuples like (foo_value, bar_value, baz_sum_value, quux_max_value).

If a group is relational field, its corresponding value will be a recordset (with a correct prefetch set that includes all returned values). For other fields, the returned values are similar to the ones you get from a record: no labels anymore, date/datetime values are not transformed strings, NULL values are returned as False, etc.

Performance

Minor performance improvements:

• The aggregate __count generates COUNT(*) instead COUNT(<table>.id) (which generates useless null checks from PostgreSQL). Moreover, COUNT(*) is only added when we explicitly ask for __count.
• Remove the implicit min(<table>.id) aggregate added previously to the query at each read_group.
• The recordsets with correct prefetch set are returned for relational group values: it adds a very small cost of instantiation, but avoids manual browsing and bad field prefetching in business code.
• Don't include extra values generated for the webclient: __domain, __range, __fold, __context.
• A falsy domain simply doesn't make any query at all, just like search. It is very convenient with domains like [(..., 'in', self.ids)] where self.ids can be empty (it happens a lot in case of onchange methods).

Performance regression:

• The values of aggregation array_agg are now ordered by id, which is less efficient but it ensures a deterministic result.

Performance Testing

Performance tests show that the new version/usage is faster in general.
With a populated database (--size=medium + some tweaks) and with all these changes,
the performance of the _read_group/read_group calls, for 2280 calls, is in average +- 9.17 % faster.

• 388 (on 2280) cases are statically slower (mean = -7.66 %, median = -5.17%) than before for several reasons:
  - more restricted access due to the commit named "simplify security of read_group", which in some case generates extra queries.
  - small overhead to instantiate recordsets;
  - array_agg is now deterministic;
  - noise.
• 862 cases are statically faster (mean = +181.08 %, median = +15.82%) than before.

Next step

The second part of this task will be focused on the API of public method read_group():

• Make it more consistent with this new _read_group.
• Simplify the parsing of data (always return raw values aside the labelled ones).
• Avoid useless aggregations.

Enterprise: https://github.com/odoo/enterprise/pull/38639

[robodoo]

Pull request status dashboard.

[jjscarafia]

Uou! Thanks a lot @ryv-odoo !